2021-10-04T01:24:38.551 INFO:root:teuthology version: 1.1.0-f08b273e 2021-10-04T01:24:38.552 INFO:teuthology.results:subset: '18738/100000' 2021-10-04T01:24:38.553 INFO:teuthology.results:seed: '7853' 2021-10-04T02:12:38.082 INFO:root:teuthology version: 1.1.0-f08b273e 2021-10-04T02:12:38.087 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-10-04T03:05:41.595 INFO:teuthology.results:Tests finished! gathering results... 2021-10-04T03:05:41.957 INFO:scrape:Found 15 jobs 2021-10-04T03:05:42.031 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413398/teuthology.log 2021-10-04T03:05:42.032 INFO:scrape:Found 3 distinct failure reasons 2021-10-04T03:05:42.033 INFO:scrape:Failure: Command failed on gibba015 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-7947-g84379239-1focal cephadm=17.0.0-7947-g84379239-1focal ceph-mds=17.0.0-7947-g84379239-1focal ceph-mgr=17.0.0-7947-g84379239-1focal ceph-common=17.0.0-7947-g84379239-1focal ceph-fuse=17.0.0-7947-g84379239-1focal ceph-test=17.0.0-7947-g84379239-1focal ceph-volume=17.0.0-7947-g84379239-1focal radosgw=17.0.0-7947-g84379239-1focal python3-rados=17.0.0-7947-g84379239-1focal python3-rgw=17.0.0-7947-g84379239-1focal python3-cephfs=17.0.0-7947-g84379239-1focal python3-rbd=17.0.0-7947-g84379239-1focal libcephfs2=17.0.0-7947-g84379239-1focal libcephfs-dev=17.0.0-7947-g84379239-1focal librados2=17.0.0-7947-g84379239-1focal librbd1=17.0.0-7947-g84379239-1focal rbd-fuse=17.0.0-7947-g84379239-1focal' 2021-10-04T03:05:42.033 INFO:scrape:3 jobs: ['6413394', '6413402', '6413390'] 2021-10-04T03:05:42.034 INFO:scrape:suites intersection: ['powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-all-distro/ubuntu_latest', 'thrashosds-health', 'whitelist_health}'] 2021-10-04T03:05:42.034 INFO:scrape:suites union: ['objectstore/bluestore-comp-snappy', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-all-distro/ubuntu_latest', 'tasks/cfuse_workunit_suites_fsync', 'tasks/cfuse_workunit_suites_truncate_delay', 'tasks/snaps-few-objects', 'thrashosds-health', 'whitelist_health}'] 2021-10-04T03:05:42.034 INFO:scrape: 2021-10-04T03:05:42.035 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on gibba005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=32d79d7a55c8d5423ad3170cd67fd9514717ab17 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2021-10-04T03:05:42.035 INFO:scrape:1 jobs: ['6413399'] 2021-10-04T03:05:42.036 INFO:scrape:suites: ['objectstore/bluestore-comp-snappy', 'powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-all-distro/centos_8.stream', 'tasks/cfuse_workunit_suites_ffsb', 'thrashosds-health', 'whitelist_health}'] 2021-10-04T03:05:42.036 INFO:scrape: 2021-10-04T03:05:42.036 INFO:scrape:Dead: None 2021-10-04T03:05:42.037 INFO:scrape:2 jobs: ['6413398', '6413391'] 2021-10-04T03:05:42.037 INFO:scrape: 2021-10-04T03:05:42.038 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba ================================================================= info: http://pulpito.ceph.com/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/ failed: 3 dead: 3 running: 0 waiting: 0 queued: 0 passed: 9 Fail ================================================================= [6413390] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-comp-snappy powercycle/default supported-all-distro/ubuntu_latest tasks/cfuse_workunit_suites_truncate_delay thrashosds-health whitelist_health} ----------------------------------------------------------------- time: 00:02:21 info: http://pulpito.ceph.com/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413390/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413390/ sentry: https://sentry.ceph.com/organizations/ceph/?query=472e7879c34e4a43b23d7c433cc9f4c4 Command failed on gibba015 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::=" --force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-7947-g84379239-1focal cephadm=17.0.0-7947-g84379239-1focal ceph-mds=17.0.0-7947-g84379239-1focal ceph-mgr=17.0.0-7947-g84379239-1focal ceph-common=17.0.0-7947-g84379239-1focal ceph- fuse=17.0.0-7947-g84379239-1focal ceph-test=17.0.0-7947-g84379239-1focal ceph-volume=17.0.0-7947-g84379239-1focal radosgw=17.0.0-7947-g84379239-1focal python3-rados=17.0.0-7947-g84379239-1focal python3-rgw=17.0.0-7947-g84379239-1focal python3-cephfs=17.0.0-7947-g84379239-1focal python3-rbd=17.0.0-7947-g84379239-1focal libcephfs2=17.0.0-7947-g84379239-1focal libcephfs- dev=17.0.0-7947-g84379239-1focal librados2=17.0.0-7947-g84379239-1focal librbd1=17.0.0-7947-g84379239-1focal rbd-fuse=17.0.0-7947-g84379239-1focal' [6413394] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-low-osd-mem-target powercycle/default supported-all-distro/ubuntu_latest tasks/snaps-few-objects thrashosds-health whitelist_health} ----------------------------------------------------------------- time: 00:02:21 info: http://pulpito.ceph.com/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413394/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413394/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5125fafe64ff40dcb3d77bb2c980d513 Command failed on gibba015 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::=" --force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-7947-g84379239-1focal cephadm=17.0.0-7947-g84379239-1focal ceph-mds=17.0.0-7947-g84379239-1focal ceph-mgr=17.0.0-7947-g84379239-1focal ceph-common=17.0.0-7947-g84379239-1focal ceph- fuse=17.0.0-7947-g84379239-1focal ceph-test=17.0.0-7947-g84379239-1focal ceph-volume=17.0.0-7947-g84379239-1focal radosgw=17.0.0-7947-g84379239-1focal python3-rados=17.0.0-7947-g84379239-1focal python3-rgw=17.0.0-7947-g84379239-1focal python3-cephfs=17.0.0-7947-g84379239-1focal python3-rbd=17.0.0-7947-g84379239-1focal libcephfs2=17.0.0-7947-g84379239-1focal libcephfs- dev=17.0.0-7947-g84379239-1focal librados2=17.0.0-7947-g84379239-1focal librbd1=17.0.0-7947-g84379239-1focal rbd-fuse=17.0.0-7947-g84379239-1focal' [6413402] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-hybrid powercycle/default supported-all-distro/ubuntu_latest tasks/cfuse_workunit_suites_fsync thrashosds-health whitelist_health} ----------------------------------------------------------------- time: 00:02:20 info: http://pulpito.ceph.com/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413402/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413402/ sentry: https://sentry.ceph.com/organizations/ceph/?query=22c32a8feb1241c4a9a14cec63f06fe9 Command failed on gibba031 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::=" --force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-7947-g84379239-1focal cephadm=17.0.0-7947-g84379239-1focal ceph-mds=17.0.0-7947-g84379239-1focal ceph-mgr=17.0.0-7947-g84379239-1focal ceph-common=17.0.0-7947-g84379239-1focal ceph- fuse=17.0.0-7947-g84379239-1focal ceph-test=17.0.0-7947-g84379239-1focal ceph-volume=17.0.0-7947-g84379239-1focal radosgw=17.0.0-7947-g84379239-1focal python3-rados=17.0.0-7947-g84379239-1focal python3-rgw=17.0.0-7947-g84379239-1focal python3-cephfs=17.0.0-7947-g84379239-1focal python3-rbd=17.0.0-7947-g84379239-1focal libcephfs2=17.0.0-7947-g84379239-1focal libcephfs- dev=17.0.0-7947-g84379239-1focal librados2=17.0.0-7947-g84379239-1focal librbd1=17.0.0-7947-g84379239-1focal rbd-fuse=17.0.0-7947-g84379239-1focal' Dead ================================================================= [6413391] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-comp-zlib powercycle/default supported-all-distro/centos_8.stream tasks/rados_api_tests thrashosds-health whitelist_health} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413391/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413391/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6413398] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-comp-lz4 powercycle/default supported-all-distro/ubuntu_latest tasks/cfuse_workunit_misc thrashosds-health whitelist_health} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413398/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413398/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6413399] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-comp-snappy powercycle/default supported-all-distro/centos_8.stream tasks/cfuse_workunit_suites_ffsb thrashosds-health whitelist_health} ----------------------------------------------------------------- time: 00:54:56 info: http://pulpito.ceph.com/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413399/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413399/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7f1af52be647435ca89af53728817c10 Command failed (workunit test suites/ffsb.sh) on gibba005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=32d79d7a55c8d5423ad3170cd67fd9514717ab17 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' Pass ================================================================= [6413388] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-bitmap powercycle/default supported-all-distro/centos_8 tasks/cfuse_workunit_suites_fsync thrashosds-health whitelist_health} time: 00:13:22 info: http://pulpito.ceph.com/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413388/ [6413389] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-comp-lz4 powercycle/default supported-all-distro/rhel_8 tasks/cfuse_workunit_suites_pjd thrashosds-health whitelist_health} time: 00:13:51 info: http://pulpito.ceph.com/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413389/ [6413392] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-comp-zstd powercycle/default supported-all-distro/centos_8 tasks/radosbench thrashosds-health whitelist_health} time: 00:59:39 info: http://pulpito.ceph.com/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413392/ [6413393] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-hybrid powercycle/default supported-all-distro/rhel_8 tasks/readwrite thrashosds-health whitelist_health} time: 00:14:41 info: http://pulpito.ceph.com/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413393/ [6413395] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-stupid powercycle/default supported-all-distro/centos_8.stream tasks/snaps-many-objects thrashosds-health whitelist_health} time: 00:26:27 info: http://pulpito.ceph.com/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413395/ [6413396] powercycle/osd/{clusters/3osd-1per-target objectstore/filestore-xfs powercycle/default supported-all-distro/centos_8 tasks/admin_socket_objecter_requests thrashosds-health whitelist_health} time: 00:12:46 info: http://pulpito.ceph.com/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413396/ [6413397] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-bitmap powercycle/default supported-all-distro/rhel_8 tasks/cfuse_workunit_kernel_untar_build thrashosds-health whitelist_health} time: 00:34:32 info: http://pulpito.ceph.com/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413397/ [6413400] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-comp-zlib powercycle/default supported-all-distro/centos_8 tasks/cfuse_workunit_suites_fsstress thrashosds-health whitelist_health} time: 00:13:58 info: http://pulpito.ceph.com/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413400/ [6413401] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-comp-zstd powercycle/default supported-all-distro/rhel_8 tasks/cfuse_workunit_suites_fsx thrashosds-health whitelist_health} time: 00:22:22 info: http://pulpito.ceph.com/teuthology-2021-09-29_11:15:03-powercycle-master-distro-basic-gibba/6413401/