2023-05-19T21:57:06.171 INFO:root:teuthology version: 0.0.1.dev100+g01abd1e
2023-05-19T21:57:06.172 INFO:teuthology.results:seed: '4501'
2023-05-19T22:01:29.758 INFO:root:teuthology version: 0.0.1.dev100+g01abd1e
2023-05-19T22:01:29.761 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-05-19T22:27:31.869 INFO:teuthology.results:Tests finished! gathering results...
2023-05-19T22:27:32.217 INFO:scrape:Found 8 jobs
2023-05-19T22:27:32.347 INFO:scrape:Found 1 distinct failure reasons
2023-05-19T22:27:32.348 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi110 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=486d4f204735218554e43161c1e7cea7781e2fdd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_latency.sh'
2023-05-19T22:27:32.348 INFO:scrape:8 jobs: ['7280699', '7280695', '7280701', '7280696', '7280700', '7280697', '7280698', '7280702']
2023-05-19T22:27:32.349 INFO:scrape:suites intersection: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'openstack}', 'rbd/mirror/{base/install', 'workloads/rbd-mirror-snapshot-workunit-latency}']
2023-05-19T22:27:32.349 INFO:scrape:suites union: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_20.04}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-workunit-latency}']
2023-05-19T22:27:32.349 INFO:scrape:
2023-05-19T22:27:32.350 INFO:teuthology.results:Sending results to choffman@redhat.com: Test Run: choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi/
failed:  8
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7280695]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:13:27
info:   http://pulpito.ceph.com/choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280695/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280695/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0370775f6fd54d7abd6d809724367685

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi052 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=486d4f204735218554e43161c1e7cea7781e2fdd
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7280696]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:13:31
info:   http://pulpito.ceph.com/choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280696/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280696/
sentry: https://sentry.ceph.com/organizations/ceph/?query=40a97326cd1f4aa49bf4f70c8baa8bd9

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi150 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=486d4f204735218554e43161c1e7cea7781e2fdd
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7280697]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_20.04} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:11:35
info:   http://pulpito.ceph.com/choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280697/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280697/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8090103da79f4e49818f8679251f302f

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi195 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=486d4f204735218554e43161c1e7cea7781e2fdd
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7280698]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:18:23
info:   http://pulpito.ceph.com/choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280698/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280698/
sentry: https://sentry.ceph.com/organizations/ceph/?query=359b6823842145f198260f0f9060a7a4

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi134 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=486d4f204735218554e43161c1e7cea7781e2fdd
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7280699]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:12:40
info:   http://pulpito.ceph.com/choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280699/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280699/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d9e2b4c62a88418d99b4992b3d8bec7a

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi110 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=486d4f204735218554e43161c1e7cea7781e2fdd
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7280700]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:16:11
info:   http://pulpito.ceph.com/choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280700/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280700/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a35354c49d2547fd9c71e8344d75f270

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi173 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=486d4f204735218554e43161c1e7cea7781e2fdd
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7280701]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:12:27
info:   http://pulpito.ceph.com/choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280701/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280701/
sentry: https://sentry.ceph.com/organizations/ceph/?query=06663deea199474eb15d6fb22ddd5364

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi159 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=486d4f204735218554e43161c1e7cea7781e2fdd
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7280702]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:13:26
info:   http://pulpito.ceph.com/choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280702/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-19_21:56:53-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280702/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5ee496a6445e4f4db24e8a5d72269c13

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi191 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=486d4f204735218554e43161c1e7cea7781e2fdd
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'