2023-05-19T21:06:27.408 INFO:root:teuthology version: 0.0.1.dev100+g01abd1e
2023-05-19T21:06:27.409 INFO:teuthology.results:seed: '5393'
2023-05-19T21:13:07.863 INFO:root:teuthology version: 0.0.1.dev100+g01abd1e
2023-05-19T21:13:07.866 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-05-19T21:44:15.440 INFO:teuthology.results:Tests finished! gathering results...
2023-05-19T21:44:15.764 INFO:scrape:Found 8 jobs
2023-05-19T21:44:16.000 INFO:scrape:Found 2 distinct failure reasons
2023-05-19T21:44:16.001 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi096 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ed413c3597d714f28526dffed215797ef97341cd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_latency.sh'
2023-05-19T21:44:16.001 INFO:scrape:7 jobs: ['7280685', '7280692', '7280687', '7280690', '7280689', '7280688', '7280686']
2023-05-19T21:44:16.001 INFO:scrape:suites intersection: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'openstack}', 'rbd/mirror/{base/install', 'workloads/rbd-mirror-snapshot-workunit-latency}']
2023-05-19T21:44:16.002 INFO:scrape:suites union: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_20.04}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-workunit-latency}']
2023-05-19T21:44:16.002 INFO:scrape:
2023-05-19T21:44:16.003 INFO:scrape:Failure: Command failed on smithi190 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/chrisphoffman/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout ed413c3597d714f28526dffed215797ef97341cd'
2023-05-19T21:44:16.003 INFO:scrape:1 jobs: ['7280691']
2023-05-19T21:44:16.003 INFO:scrape:suites: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd-mirror-snapshot-workunit-latency}']
2023-05-19T21:44:16.004 INFO:scrape:
2023-05-19T21:44:16.005 INFO:teuthology.results:Sending results to choffman@redhat.com: Test Run: choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi/
failed:  8
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7280685]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:14:01
info:   http://pulpito.ceph.com/choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280685/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280685/
sentry: https://sentry.ceph.com/organizations/ceph/?query=843f82e1a08c40e19a96c24d4e2e030b

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi096 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ed413c3597d714f28526dffed215797ef97341cd
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7280686]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:13:47
info:   http://pulpito.ceph.com/choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280686/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280686/
sentry: https://sentry.ceph.com/organizations/ceph/?query=85d620e667f34d64bd6455f87d6930b0

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi052 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ed413c3597d714f28526dffed215797ef97341cd
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7280687]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_20.04} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:11:38
info:   http://pulpito.ceph.com/choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280687/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280687/
sentry: https://sentry.ceph.com/organizations/ceph/?query=95b63d02f3be42a9a04bee29b96ef9c1

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi098 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ed413c3597d714f28526dffed215797ef97341cd
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7280688]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:19:27
info:   http://pulpito.ceph.com/choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280688/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280688/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bba61414ef6b4c5481834400d98a877b

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi195 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ed413c3597d714f28526dffed215797ef97341cd
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7280689]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:19:19
info:   http://pulpito.ceph.com/choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280689/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280689/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d156656610354dbc831b2bff5a04ad07

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi188 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ed413c3597d714f28526dffed215797ef97341cd
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7280690]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:13:30
info:   http://pulpito.ceph.com/choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280690/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280690/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d59b9775401941ec98abf8f118fb990f

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi136 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ed413c3597d714f28526dffed215797ef97341cd
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'

[7280691]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:20:23
info:   http://pulpito.ceph.com/choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280691/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280691/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8e5305c9a7cc4a27a2d7cb0deedaef8d

    Command failed on smithi190 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone
    https://github.com/chrisphoffman/ceph.git
    /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd
    /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout
    ed413c3597d714f28526dffed215797ef97341cd'

[7280692]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-latency}
-----------------------------------------------------------------
time:   00:17:19
info:   http://pulpito.ceph.com/choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280692/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-05-19_21:05:43-rbd-wip-choffman-rbd-59728-distro-default-smithi/7280692/
sentry: https://sentry.ceph.com/organizations/ceph/?query=16ad8af32276456796ede347fcde4295

    Command failed (workunit test rbd/rbd_mirror_latency.sh) on smithi110 with
    status 127: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ed413c3597d714f28526dffed215797ef97341cd
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror LATENCY=30ms adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /ho
    me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror
    _latency.sh'