Found 447 jobs Missing teuthology log /home/teuthworker/archive/teuthology-2021-04-14_05:07:03-powercycle-nautilus-distro-basic-gibba/6046159/teuthology.log Found 7 distinct failure reasons Dead: None 7 jobs: ['6046159', '6046321', '6046171', '6046359', '6046520', '6046172', '6046276'] Failure: 'NoneType' object has no attribute 'get' 2 jobs: ['6046556', '6046185'] suites intersection: ['powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-all-distro/centos_latest', 'thrashosds-health', 'whitelist_health}'] suites union: ['objectstore/bluestore-comp-zlib', 'objectstore/bluestore-stupid', 'powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-all-distro/centos_latest', 'tasks/cfuse_workunit_suites_fsync', 'tasks/snaps-many-objects', 'thrashosds-health', 'whitelist_health}'] Failure: reached maximum tries (500) after waiting for 3000 seconds 8 jobs: ['6046147', '6046427', '6046259', '6046532', '6046203', '6046483', '6046371', '6046539'] suites intersection: ['powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'thrashosds-health', 'whitelist_health}'] suites union: ['objectstore/bluestore-hybrid', 'objectstore/bluestore-stupid', 'powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-all-distro/centos_latest', 'supported-all-distro/rhel_7', 'supported-all-distro/ubuntu_16.04', 'supported-all-distro/ubuntu_latest', 'tasks/cfuse_workunit_suites_ffsb', 'tasks/radosbench', 'thrashosds-health', 'whitelist_health}'] Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2 jobs: ['6046454', '6046160'] suites intersection: ['powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'tasks/rados_api_tests', 'thrashosds-health', 'whitelist_health}'] suites union: ['objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-all-distro/ubuntu_16.04', 'supported-all-distro/ubuntu_latest', 'tasks/rados_api_tests', 'thrashosds-health', 'whitelist_health}'] Failure: Command failed (workunit test suites/ffsb.sh) on gibba003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=533a855677b2d240e650ac0a2ccad7ad00249ae5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 1 jobs: ['6046448'] suites: ['objectstore/bluestore-comp-zlib', 'powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-all-distro/centos_latest', 'tasks/cfuse_workunit_suites_ffsb', 'thrashosds-health', 'whitelist_health}'] Crash: Command failed (workunit test suites/ffsb.sh) on gibba013 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=533a855677b2d240e650ac0a2ccad7ad00249ae5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' ceph version 14.2.19-115-g533a855677 (533a855677b2d240e650ac0a2ccad7ad00249ae5) nautilus (stable) 1: (()+0x12980) [0x7f4375df6980] 2: (gsignal()+0xc7) [0x7f437548ffb7] 3: (abort()+0x141) [0x7f4375491921] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1aa) [0x7f43762e3758] 5: (()+0x2e08e7) [0x7f43762e38e7] 6: (ObjectCacher::bh_write_commit(long, sobject_t, std::vector, std::allocator > >&, unsigned long, int)+0x1839) [0x559d238556e9] 7: (ObjectCacher::C_WriteCommit::finish(int)+0x6c) [0x559d2385996c] 8: (Context::complete(int)+0x9) [0x559d237f4ac9] 9: (C_Lock::finish(int)+0x29) [0x559d237f5909] 10: (Context::complete(int)+0x9) [0x559d237f4ac9] 11: (Finisher::finisher_thread_entry()+0x15e) [0x7f4376385c8e] 12: (()+0x76db) [0x7f4375deb6db] 13: (clone()+0x3f) [0x7f437557271f] 2 jobs: ['6046168', '6046434'] suites intersection: ['powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-all-distro/ubuntu_latest', 'tasks/cfuse_workunit_suites_ffsb', 'thrashosds-health', 'whitelist_health}'] suites union: ['objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-all-distro/ubuntu_latest', 'tasks/cfuse_workunit_suites_ffsb', 'thrashosds-health', 'whitelist_health}'] Failure: failed to install new kernel version within timeout 1 jobs: ['6046237'] suites: ['objectstore/bluestore-bitmap', 'powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-all-distro/ubuntu_latest', 'tasks/cfuse_workunit_misc', 'thrashosds-health', 'whitelist_health}']