2023-07-02T15:06:51.939 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-02T15:06:51.940 INFO:teuthology.results:subset: '25/99' 2023-07-02T15:06:51.940 INFO:teuthology.results:seed: '3615' 2023-07-02T15:08:00.258 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-02T15:08:00.261 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-07-02T15:45:03.112 INFO:teuthology.results:Tests finished! gathering results... 2023-07-02T15:45:03.374 INFO:scrape:Found 10 jobs 2023-07-02T15:45:04.485 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324408/teuthology.log 2023-07-02T15:45:07.440 INFO:scrape:Found 2 distinct failure reasons 2023-07-02T15:45:07.441 INFO:scrape:Failure: Command failed (workunit test rbd/diff_continuous.sh) on smithi164 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' 2023-07-02T15:45:07.441 INFO:scrape:9 jobs: ['7324406', '7324405', '7324402', '7324411', '7324409', '7324410', '7324403', '7324407', '7324404'] 2023-07-02T15:45:07.442 INFO:scrape:suites intersection: ['cluster/{fixed-3', 'msgr-failures/few', 'openstack}', 'rbd/nbd/{base/install', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashosds-health', 'workloads/rbd_nbd_diff_continuous}'] 2023-07-02T15:45:07.442 INFO:scrape:suites union: ['cluster/{fixed-3', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-hybrid', 'openstack}', 'rbd/nbd/{base/install', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashosds-health', 'workloads/rbd_nbd_diff_continuous}'] 2023-07-02T15:45:07.442 INFO:scrape: 2023-07-02T15:45:07.442 INFO:scrape:Dead: None 2023-07-02T15:45:07.443 INFO:scrape:1 jobs: ['7324408'] 2023-07-02T15:45:07.443 INFO:scrape: 2023-07-02T15:45:07.444 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/ failed: 9 dead: 1 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7324402] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:18:59 info: http://pulpito.ceph.com/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324402/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324402/ sentry: https://sentry.ceph.com/organizations/ceph/?query=629e5147f8c34170898d1c429363468b Command failed (workunit test rbd/diff_continuous.sh) on smithi141 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' [7324403] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:20:02 info: http://pulpito.ceph.com/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324403/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324403/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1e27459651d74546b73e2174ce1abd1f Command failed (workunit test rbd/diff_continuous.sh) on smithi111 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' [7324404] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:18:47 info: http://pulpito.ceph.com/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324404/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324404/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1bc55ec5b48d41618aca69b55a6935ea Command failed (workunit test rbd/diff_continuous.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' [7324405] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:19:41 info: http://pulpito.ceph.com/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324405/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324405/ sentry: https://sentry.ceph.com/organizations/ceph/?query=579a4a9a722e42408d098bdcc3c5e53a Command failed (workunit test rbd/diff_continuous.sh) on smithi196 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' [7324406] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:20:54 info: http://pulpito.ceph.com/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324406/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324406/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7b7806f6f1954f92abfe0e9fd3121c57 Command failed (workunit test rbd/diff_continuous.sh) on smithi164 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' [7324407] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:18:36 info: http://pulpito.ceph.com/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324407/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324407/ sentry: https://sentry.ceph.com/organizations/ceph/?query=53c68531af1640bfaab1529788f3f835 Command failed (workunit test rbd/diff_continuous.sh) on smithi192 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' [7324409] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:21:52 info: http://pulpito.ceph.com/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324409/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324409/ sentry: https://sentry.ceph.com/organizations/ceph/?query=29d37940111a4f26977bb1f815a0028c Command failed (workunit test rbd/diff_continuous.sh) on smithi174 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' [7324410] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:22:22 info: http://pulpito.ceph.com/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324410/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324410/ sentry: https://sentry.ceph.com/organizations/ceph/?query=224a41e5453743259edd72ead77193ba Command failed (workunit test rbd/diff_continuous.sh) on smithi167 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' [7324411] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:20:06 info: http://pulpito.ceph.com/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324411/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324411/ sentry: https://sentry.ceph.com/organizations/ceph/?query=95eb2bec77db4a85ac5e1951c8e1065e Command failed (workunit test rbd/diff_continuous.sh) on smithi130 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' Dead ================================================================= [7324408] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324408/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:06:33-rbd-quincy-distro-default-smithi/7324408/ Error reimaging machines: Failed to power on smithi062