Found 220 jobs Found 6 distinct failure reasons Failure: "2021-01-23T16:51:22.914831+0000 osd.3 (osd.3) 3 : cluster [WRN] slow request osd_op(client.4349.0:349407 2.2 2:5992b234:::rbd_data.112f56adf633.0000000000000a0d:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3166208~12288 in=12288b] snapc 0=[] ondisk+write+known_if_redirected e21) initiated 2021-01-23T16:50:52.638050+0000 currently waiting for sub ops" in cluster log 2 jobs: ['5819964', '5820076'] suites intersection: ['clusters/fixed-3', 'conf', 'krbd/rbd/{bluestore-bitmap', 'msgr-failures/many'] suites union: ['clusters/fixed-3', 'conf', 'krbd/rbd/{bluestore-bitmap', 'ms_mode/crc', 'ms_mode/legacy', 'msgr-failures/many', 'tasks/rbd_workunit_suites_ffsb}', 'tasks/rbd_workunit_suites_fsstress}'] Failure: Command failed on smithi135 with status 3: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludemxb_b516 -r -- '-g auto -g blockdev -x clone'" 1 jobs: ['5819951'] suites: ['conf', 'krbd/singleton/{bluestore-bitmap', 'ms_mode$/{prefer-crc}', 'msgr-failures/few', 'tasks/rbd_xfstests}'] Failure: "2021-01-23T19:37:55.614153+0000 osd.3 (osd.3) 3 : cluster [WRN] slow request osd_op(client.6863.0:23400 2.5 2:a655690b:::rbd_header.143a20e10beb:head [watch reconnect cookie 18446462598732841009 gen 109] snapc 0=[] ondisk+write+known_if_redirected e21) initiated 2021-01-23T19:37:25.419918+0000 currently started" in cluster log 1 jobs: ['5820147'] suites: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'ms_mode/legacy', 'msgr-failures/many', 'tasks/krbd_udev_enumerate}'] Failure: Found coredumps on ubuntu@smithi175.front.sepia.ceph.com 1 jobs: ['5820120'] suites: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'ms_mode/crc', 'msgr-failures/few', 'tasks/rbd_concurrent}'] Dead: b"TypeError: argument of type 'NoneType' is not iterable" ceph version 16.0.0-9153-g97717fdd (97717fdd03f2bff5c2adf3b4cd831ab3c629fcd8) pacific (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fd287493980] 2: pthread_cond_wait() 3: (std::condition_variable::wait(std::unique_lock&)+0xc) [0x7fd286b658bc] 4: (Throttle::_wait(long, std::unique_lock&)+0x115) [0x55e754744cf5] 5: (Throttle::get(long, long)+0xb6) [0x55e7547456d6] 6: (BlueStore::BlueStoreThrottle::finish_start_transaction(KeyValueDB&, BlueStore::TransContext&, std::chrono::time_point > >)+0x32) [0x55e754581c22] 7: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x906) [0x55e7546189e6] 8: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x54) [0x55e754247604] 9: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0xc32) [0x55e754444b22] 10: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xb6e) [0x55e7541cbbee] 11: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xeaa) [0x55e7542258ea] 12: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x4006) [0x55e75422a5a6] 13: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xcc7) [0x55e754235aa7] 14: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x17b) [0x55e7540bc37b] 15: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x6a) [0x55e75431d2ba] 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xcd5) [0x55e7540da1c5] 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x55e75474ed7c] 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55e754752030] 19: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fd2874886db] 20: clone() 1 jobs: ['5820062'] suites: ['conf', 'krbd/singleton/{bluestore-bitmap', 'ms_mode$/{legacy}', 'msgr-failures/many', 'tasks/rbd_xfstests}'] Failure: psutil.NoSuchProcess process no longer exists (pid=15066) 1 jobs: ['5819952'] suites: ['ceph/ceph', 'clusters/fixed-3', 'conf', 'krbd/thrash/{bluestore-bitmap', 'ms_mode$/{crc}', 'thrashers/backoff', 'thrashosds-health', 'workloads/rbd_fio}']