2021-05-25T06:40:52.719 INFO:root:teuthology version: 1.1.0-d92ec140 2021-05-25T06:40:52.722 INFO:teuthology.results:seed: '8793' 2021-05-25T06:42:11.640 INFO:root:teuthology version: 1.1.0-d92ec140 2021-05-25T06:42:11.645 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-05-25T18:42:59.476 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2021-05-25T18:42:59.988 INFO:scrape:Found 5 jobs 2021-05-25T18:43:01.119 INFO:scrape:Found 3 distinct failure reasons 2021-05-25T18:43:01.120 INFO:scrape:Failure: Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) 2021-05-25T18:43:01.120 INFO:scrape:2 jobs: ['6133764', '6133761'] 2021-05-25T18:43:01.120 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'fs:functional:/{begin.yaml', 'mds.yaml', 'mon.yaml', 'no_client_pidfile.yaml', 'osd.yaml}', 'overrides/{frag_enable.yaml', 'tasks/mds-full.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}'] 2021-05-25T18:43:01.121 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'distro/{centos_8.yaml}', 'distro/{rhel_8.yaml}', 'fs:functional:/{begin.yaml', 'mds.yaml', 'mon.yaml', 'mount/fuse.yaml', 'mount/kclient/{mount.yaml', 'ms-die-on-skipped.yaml}}', 'no_client_pidfile.yaml', 'objectstore/bluestore-bitmap.yaml', 'objectstore/bluestore-ec-root.yaml', 'osd.yaml}', 'overrides/{distro/stock/{k-stock.yaml', 'overrides/{frag_enable.yaml', 'rhel_8.yaml}', 'tasks/mds-full.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}'] 2021-05-25T18:43:01.121 INFO:scrape: 2021-05-25T18:43:01.122 INFO:scrape:Crash: Test failure: test_full_fclose (tasks.cephfs.test_full.TestClusterFull) 2021-05-25T18:43:01.122 INFO:scrape:ceph version 17.0.0-4201-g9eeed0fc (9eeed0fc410d5ee8fe1a6b50187065f5aef3dcf8) quincy (dev) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f64285beb20] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x5582ae305e05] 5: (ReplicatedBackend::_do_push(boost::intrusive_ptr)+0x198) [0x5582ae7a65a8] 6: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x2a8) [0x5582ae7a8748] 7: (PGBackend::handle_message(boost::intrusive_ptr)+0x52) [0x5582ae5defa2] 8: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x5de) [0x5582ae5808ee] 9: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x309) [0x5582ae409c49] 10: (ceph::osd::scheduler::PGRecoveryMsg::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x68) [0x5582ae66abf8] 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xa58) [0x5582ae427108] 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x5582aeab2554] 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x5582aeab38f4] 14: (Thread::_entry_func(void*)+0xd) [0x5582aea9961d] 15: /lib64/libpthread.so.0(+0x814a) [0x7f64285b414a] 16: clone() 2021-05-25T18:43:01.123 INFO:scrape:1 jobs: ['6133760'] 2021-05-25T18:43:01.123 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'distro/{rhel_8.yaml}', 'fs:functional:/{begin.yaml', 'mds.yaml', 'mon.yaml', 'mount/kclient/{mount.yaml', 'ms-die-on-skipped.yaml}}', 'no_client_pidfile.yaml', 'objectstore/bluestore-ec-root.yaml', 'osd.yaml}', 'overrides/{distro/stock/{k-stock.yaml', 'overrides/{frag_enable.yaml', 'rhel_8.yaml}', 'tasks/mds-full.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}'] 2021-05-25T18:43:01.123 INFO:scrape: 2021-05-25T18:43:01.124 INFO:scrape:Dead: b"2021-05-25T07:08:18.357 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty" 2021-05-25T18:43:01.124 INFO:scrape:ceph version 17.0.0-4201-g9eeed0fc (9eeed0fc410d5ee8fe1a6b50187065f5aef3dcf8) quincy (dev) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f8401eb7b20] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x7f8403111d41] 5: (operator<<(std::ostream&, ClientMetricType const&)+0x10e) [0x7f84033c857e] 6: (MClientMetrics::print(std::ostream&) const+0x1b1) [0x7f84033c8761] 7: (DispatchQueue::pre_dispatch(boost::intrusive_ptr const&)+0x710) [0x7f8403379ef0] 8: (DispatchQueue::fast_dispatch(boost::intrusive_ptr const&)+0x32) [0x7f840337be62] 9: (ProtocolV1::handle_message_footer(char*, int)+0x1500) [0x7f84034387e0] 10: /usr/lib64/ceph/libceph-common.so.2(+0x59ae0d) [0x7f840342fe0d] 11: (AsyncConnection::process()+0x789) [0x7f840341e269] 12: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0x1547) [0x7f840347aab7] 13: /usr/lib64/ceph/libceph-common.so.2(+0x5eca11) [0x7f8403481a11] 14: /lib64/libstdc++.so.6(+0xc2ba3) [0x7f84012f0ba3] 15: /lib64/libpthread.so.0(+0x814a) [0x7f8401ead14a] 16: clone() 2021-05-25T18:43:01.125 INFO:scrape:2 jobs: ['6133763', '6133762'] 2021-05-25T18:43:01.125 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'distro/{centos_8.yaml}', 'fs:functional:/{begin.yaml', 'k-testing.yaml}', 'mds.yaml', 'mon.yaml', 'mount/kclient/{mount.yaml', 'ms-die-on-skipped.yaml}}', 'no_client_pidfile.yaml', 'objectstore/bluestore-ec-root.yaml', 'osd.yaml}', 'overrides/{frag_enable.yaml', 'tasks/mds-full.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}'] 2021-05-25T18:43:01.126 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'distro/{centos_8.yaml}', 'fs:functional:/{begin.yaml', 'k-testing.yaml}', 'mds.yaml', 'mon.yaml', 'mount/kclient/{mount.yaml', 'ms-die-on-skipped.yaml}}', 'no_client_pidfile.yaml', 'objectstore/bluestore-ec-root.yaml', 'osd.yaml}', 'overrides/{distro/testing/{flavor/centos_latest.yaml', 'overrides/{distro/testing/{flavor/ubuntu_latest.yaml', 'overrides/{frag_enable.yaml', 'tasks/mds-full.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}'] 2021-05-25T18:43:01.126 INFO:scrape: