2023-05-22T15:19:57.682 INFO:root:teuthology version: 0.0.1.dev100+g01abd1e
2023-05-22T15:19:57.683 INFO:teuthology.results:seed: '3598'
2023-05-22T15:24:30.474 INFO:root:teuthology version: 0.0.1.dev100+g01abd1e
2023-05-22T15:24:30.477 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-05-22T15:54:32.960 INFO:teuthology.results:Tests finished! gathering results...
2023-05-22T15:54:33.394 INFO:scrape:Found 8 jobs
2023-05-22T15:54:34.348 INFO:scrape:Found 1 distinct failure reasons
2023-05-22T15:54:34.349 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi100 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ea3330e3f22798e0042fb53083b727d8ee2b90f5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_latency.sh'
2023-05-22T15:54:34.349 INFO:scrape:8 jobs: ['7282652', '7282649', '7282653', '7282654', '7282651', '7282655', '7282648', '7282650']
2023-05-22T15:54:34.350 INFO:scrape:suites intersection: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'openstack}', 'rbd/mirror/{base/install', 'workloads/rbd-mirror-snapshot-workunit-latency}']
2023-05-22T15:54:34.350 INFO:scrape:suites union: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_20.04}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-workunit-latency}']
2023-05-22T15:54:34.350 INFO:scrape:
2023-05-22T15:54:34.351 INFO:teuthology.results:Sending results to choffman@redhat.com: Test Run: choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi/
failed:  8
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7282648]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_20.04} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:12:04
info:   http://pulpito.ceph.com/choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi/7282648/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi/7282648/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ac6ed79e65464805b62e02945b7d604a

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi139 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ea3330e3f22798e0042fb53083b727d8ee2b90f5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7282649]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:13:56
info:   http://pulpito.ceph.com/choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi/7282649/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi/7282649/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2dc2e6c2605a459ca467c4a21a359579

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi141 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ea3330e3f22798e0042fb53083b727d8ee2b90f5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7282650]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:15:55
info:   http://pulpito.ceph.com/choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi/7282650/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi/7282650/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9150e9a6bf1b4b02aa8bbc052630a1ab

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi145 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ea3330e3f22798e0042fb53083b727d8ee2b90f5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7282651]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:13:12
info:   http://pulpito.ceph.com/choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi/7282651/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi/7282651/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e049512c74f849ff926910df4b36f5ae

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi191 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ea3330e3f22798e0042fb53083b727d8ee2b90f5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7282652]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:13:39
info:   http://pulpito.ceph.com/choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi/7282652/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi/7282652/
sentry: https://sentry.ceph.com/organizations/ceph/?query=186b445bf627445391e367f948d144cf

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi100 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ea3330e3f22798e0042fb53083b727d8ee2b90f5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7282653]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:25:22
info:   http://pulpito.ceph.com/choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi/7282653/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi/7282653/
sentry: https://sentry.ceph.com/organizations/ceph/?query=623dabe938774a4d9ec7475af468436d

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi151 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ea3330e3f22798e0042fb53083b727d8ee2b90f5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7282654]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:13:27
info:   http://pulpito.ceph.com/choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi/7282654/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi/7282654/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b29818848ff54ae2a04e0255fb1f25e7

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi184 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ea3330e3f22798e0042fb53083b727d8ee2b90f5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7282655]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:13:52
info:   http://pulpito.ceph.com/choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi/7282655/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-22_15:18:04-rbd-wip-choffman-rbd-59728-distro-default-smithi/7282655/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fa5d1f775fbc404b870e918be92758b3

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi163 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ea3330e3f22798e0042fb53083b727d8ee2b90f5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'