2023-07-02T15:50:58.672 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-02T15:50:58.673 INFO:teuthology.results:subset: '25/99' 2023-07-02T15:50:58.673 INFO:teuthology.results:seed: '3615' 2023-07-02T15:51:03.391 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-02T15:51:03.392 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-07-02T16:33:06.270 INFO:teuthology.results:Tests finished! gathering results... 2023-07-02T16:33:06.566 INFO:scrape:Found 10 jobs 2023-07-02T16:33:10.348 INFO:scrape:Found 3 distinct failure reasons 2023-07-02T16:33:10.349 INFO:scrape:Failure: Command failed (workunit test rbd/diff_continuous.sh) on smithi193 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' 2023-07-02T16:33:10.350 INFO:scrape:8 jobs: ['7324438', '7324447', '7324442', '7324443', '7324444', '7324445', '7324439', '7324446'] 2023-07-02T16:33:10.350 INFO:scrape:suites intersection: ['cluster/{fixed-3', 'msgr-failures/few', 'openstack}', 'rbd/nbd/{base/install', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashosds-health', 'workloads/rbd_nbd_diff_continuous}'] 2023-07-02T16:33:10.350 INFO:scrape:suites union: ['cluster/{fixed-3', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-hybrid', 'openstack}', 'rbd/nbd/{base/install', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashosds-health', 'workloads/rbd_nbd_diff_continuous}'] 2023-07-02T16:33:10.351 INFO:scrape: 2023-07-02T16:33:10.351 INFO:scrape:Dead: b'2023-07-02T16:06:20.097 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off' 2023-07-02T16:33:10.351 INFO:scrape:1 jobs: ['7324441'] 2023-07-02T16:33:10.352 INFO:scrape:suites: ['cluster/{fixed-3', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'openstack}', 'rbd/nbd/{base/install', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashosds-health', 'workloads/rbd_nbd_diff_continuous}'] 2023-07-02T16:33:10.352 INFO:scrape: 2023-07-02T16:33:10.352 INFO:scrape:Failure: Command failed on smithi170 with status 1: 'sudo grub2-mkconfig -o /boot/grub2/grub.cfg' 2023-07-02T16:33:10.352 INFO:scrape:1 jobs: ['7324440'] 2023-07-02T16:33:10.353 INFO:scrape:suites: ['cluster/{fixed-3', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'openstack}', 'rbd/nbd/{base/install', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashosds-health', 'workloads/rbd_nbd_diff_continuous}'] 2023-07-02T16:33:10.353 INFO:scrape: 2023-07-02T16:33:10.354 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/ failed: 9 dead: 1 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7324438] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:23:40 info: http://pulpito.ceph.com/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324438/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324438/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5ff0ea1e5bb3414282dc3dff7653f674 Command failed (workunit test rbd/diff_continuous.sh) on smithi193 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' [7324439] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:24:30 info: http://pulpito.ceph.com/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324439/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324439/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a6f906b815f46ab917a06ab81401749 Command failed (workunit test rbd/diff_continuous.sh) on smithi066 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' [7324440] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324440/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324440/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9023dee5ba934c57b8df83f2bb3b6fdf Command failed on smithi170 with status 1: 'sudo grub2-mkconfig -o /boot/grub2/grub.cfg' [7324442] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:23:19 info: http://pulpito.ceph.com/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324442/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324442/ sentry: https://sentry.ceph.com/organizations/ceph/?query=53b156a0e8ea409ab2e5691b7357fce3 Command failed (workunit test rbd/diff_continuous.sh) on smithi154 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' [7324443] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:20:45 info: http://pulpito.ceph.com/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324443/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324443/ sentry: https://sentry.ceph.com/organizations/ceph/?query=219b36d6a7d84930a006e505b097e688 Command failed (workunit test rbd/diff_continuous.sh) on smithi190 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' [7324444] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:22:51 info: http://pulpito.ceph.com/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324444/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324444/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2b707f8154da4ffca714c81ac1c2a47a Command failed (workunit test rbd/diff_continuous.sh) on smithi202 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' [7324445] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:23:26 info: http://pulpito.ceph.com/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324445/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324445/ sentry: https://sentry.ceph.com/organizations/ceph/?query=62865188b2cd4c86a13a26ce0e30bd5c Command failed (workunit test rbd/diff_continuous.sh) on smithi133 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' [7324446] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:23:41 info: http://pulpito.ceph.com/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324446/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324446/ sentry: https://sentry.ceph.com/organizations/ceph/?query=35d6340ce40a4fc19347a02898bf0106 Command failed (workunit test rbd/diff_continuous.sh) on smithi142 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' [7324447] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:22:01 info: http://pulpito.ceph.com/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324447/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324447/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a9559a72fd3649f5ba341ea5c3dce9df Command failed (workunit test rbd/diff_continuous.sh) on smithi181 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c35823f8c2a4fda82db9fccfb6aa26cbf9b3ada6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' Dead ================================================================= [7324441] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324441/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-02_15:50:52-rbd-quincy-distro-default-smithi/7324441/