2024-02-13T18:05:15.857 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-13T18:05:15.858 INFO:teuthology.results:subset: '5/99'
2024-02-13T18:05:15.858 INFO:teuthology.results:seed: '3494'
2024-02-13T18:13:18.934 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-13T18:13:18.938 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-02-13T18:51:21.925 INFO:teuthology.results:Tests finished! gathering results...
2024-02-13T18:51:22.294 INFO:scrape:Found 10 jobs
2024-02-13T18:51:24.088 INFO:scrape:Found 1 distinct failure reasons
2024-02-13T18:51:24.088 INFO:scrape:Failure: Command failed (workunit test rbd/diff_continuous.sh) on smithi144 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bab43e83ad74841f19b4294dbbe2edc306759a8d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=krbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh'
2024-02-13T18:51:24.088 INFO:scrape:10 jobs: ['7558788', '7558797', '7558792', '7558793', '7558795', '7558794', '7558796', '7558790', '7558791', '7558789']
2024-02-13T18:51:24.089 INFO:scrape:suites intersection: ['ceph/ceph', 'clusters/fixed-3', 'conf', 'krbd/thrash/{bluestore-bitmap', 'thrashosds-health', 'workloads/krbd_diff_continuous}']
2024-02-13T18:51:24.089 INFO:scrape:suites union: ['ceph/ceph', 'clusters/fixed-3', 'conf', 'krbd/thrash/{bluestore-bitmap', 'ms_mode$/{crc-rxbounce}', 'ms_mode$/{legacy}', 'thrashers/mon-thrasher', 'thrashers/pggrow', 'thrashosds-health', 'workloads/krbd_diff_continuous}']
2024-02-13T18:51:24.089 INFO:scrape:
2024-02-13T18:51:24.089 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi
=================================================================
info:    https://pulpito.ceph.com/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/
failed:  10
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7558788]  krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/pggrow thrashosds-health workloads/krbd_diff_continuous}
-----------------------------------------------------------------
time:   00:26:03
info:   https://pulpito.ceph.com/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558788/
log:    http://qa-proxy.ceph.com/teuthology/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558788/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dcf76d415c18433b904a2a076b94bd4f

    Command failed (workunit test rbd/diff_continuous.sh) on smithi144 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bab43e83ad74841f19b4294dbbe2edc306759a8d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=krbd adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh'

[7558789]  krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/pggrow thrashosds-health workloads/krbd_diff_continuous}
-----------------------------------------------------------------
time:   00:26:51
info:   https://pulpito.ceph.com/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558789/
log:    http://qa-proxy.ceph.com/teuthology/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558789/
sentry: https://sentry.ceph.com/organizations/ceph/?query=438e99697cc94fe288376a468eee6ed7

    Command failed (workunit test rbd/diff_continuous.sh) on smithi152 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bab43e83ad74841f19b4294dbbe2edc306759a8d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=krbd adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh'

[7558790]  krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/pggrow thrashosds-health workloads/krbd_diff_continuous}
-----------------------------------------------------------------
time:   00:26:29
info:   https://pulpito.ceph.com/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558790/
log:    http://qa-proxy.ceph.com/teuthology/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558790/
sentry: https://sentry.ceph.com/organizations/ceph/?query=60eae0dde60e4d51a4b53cdcefcbd665

    Command failed (workunit test rbd/diff_continuous.sh) on smithi132 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bab43e83ad74841f19b4294dbbe2edc306759a8d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=krbd adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh'

[7558791]  krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/pggrow thrashosds-health workloads/krbd_diff_continuous}
-----------------------------------------------------------------
time:   00:25:40
info:   https://pulpito.ceph.com/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558791/
log:    http://qa-proxy.ceph.com/teuthology/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558791/
sentry: https://sentry.ceph.com/organizations/ceph/?query=817f8543bf86487d9fe726bd426c4185

    Command failed (workunit test rbd/diff_continuous.sh) on smithi145 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bab43e83ad74841f19b4294dbbe2edc306759a8d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=krbd adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh'

[7558792]  krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/pggrow thrashosds-health workloads/krbd_diff_continuous}
-----------------------------------------------------------------
time:   00:27:58
info:   https://pulpito.ceph.com/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558792/
log:    http://qa-proxy.ceph.com/teuthology/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558792/
sentry: https://sentry.ceph.com/organizations/ceph/?query=44e2ee87d159410598fd365a7eda2bec

    Command failed (workunit test rbd/diff_continuous.sh) on smithi162 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bab43e83ad74841f19b4294dbbe2edc306759a8d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=krbd adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh'

[7558793]  krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc-rxbounce} thrashers/mon-thrasher thrashosds-health workloads/krbd_diff_continuous}
-----------------------------------------------------------------
time:   00:26:57
info:   https://pulpito.ceph.com/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558793/
log:    http://qa-proxy.ceph.com/teuthology/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558793/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cde7c4e1f14e40958b14fee46b8b8b0a

    Command failed (workunit test rbd/diff_continuous.sh) on smithi187 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bab43e83ad74841f19b4294dbbe2edc306759a8d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=krbd adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh'

[7558794]  krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc-rxbounce} thrashers/mon-thrasher thrashosds-health workloads/krbd_diff_continuous}
-----------------------------------------------------------------
time:   00:26:58
info:   https://pulpito.ceph.com/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558794/
log:    http://qa-proxy.ceph.com/teuthology/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558794/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3ddde8d29f1a47d1bbc16c470e965a97

    Command failed (workunit test rbd/diff_continuous.sh) on smithi117 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bab43e83ad74841f19b4294dbbe2edc306759a8d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=krbd adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh'

[7558795]  krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc-rxbounce} thrashers/mon-thrasher thrashosds-health workloads/krbd_diff_continuous}
-----------------------------------------------------------------
time:   00:28:34
info:   https://pulpito.ceph.com/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558795/
log:    http://qa-proxy.ceph.com/teuthology/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558795/
sentry: https://sentry.ceph.com/organizations/ceph/?query=86afb77456854dbebb6ca1e301c655a5

    Command failed (workunit test rbd/diff_continuous.sh) on smithi183 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bab43e83ad74841f19b4294dbbe2edc306759a8d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=krbd adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh'

[7558796]  krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc-rxbounce} thrashers/mon-thrasher thrashosds-health workloads/krbd_diff_continuous}
-----------------------------------------------------------------
time:   00:25:06
info:   https://pulpito.ceph.com/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558796/
log:    http://qa-proxy.ceph.com/teuthology/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558796/
sentry: https://sentry.ceph.com/organizations/ceph/?query=254b79b5d8214b7aba3fd05a86214509

    Command failed (workunit test rbd/diff_continuous.sh) on smithi118 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bab43e83ad74841f19b4294dbbe2edc306759a8d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=krbd adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh'

[7558797]  krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc-rxbounce} thrashers/mon-thrasher thrashosds-health workloads/krbd_diff_continuous}
-----------------------------------------------------------------
time:   00:26:19
info:   https://pulpito.ceph.com/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558797/
log:    http://qa-proxy.ceph.com/teuthology/dis-2024-02-13_18:03:23-krbd-main-wip-revert-exclusive-lock-snapc-default-smithi/7558797/
sentry: https://sentry.ceph.com/organizations/ceph/?query=208971811e224cb99f6b7d3fc9c05992

    Command failed (workunit test rbd/diff_continuous.sh) on smithi157 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bab43e83ad74841f19b4294dbbe2edc306759a8d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=krbd adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh'