2021-05-26T04:58:03.176 INFO:root:teuthology version: 1.1.0-0c70841a 2021-05-26T04:58:03.178 INFO:teuthology.results:subset: '1/10' 2021-05-26T04:58:03.179 INFO:teuthology.results:seed: '4983' 2021-05-26T05:02:47.837 INFO:root:teuthology version: 1.1.0-0c70841a 2021-05-26T05:02:47.843 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-05-26T06:19:53.421 INFO:teuthology.results:Tests finished! gathering results... 2021-05-26T06:19:54.620 INFO:scrape:Found 16 jobs 2021-05-26T06:19:54.921 INFO:scrape:Found 1 distinct failure reasons 2021-05-26T06:19:54.922 INFO:scrape:Crash: Command failed on smithi118 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats' 2021-05-26T06:19:54.923 INFO:scrape:ceph version 17.0.0-4472-g5276c2a3 (5276c2a3d0eae8e5f7b7f8efd1bb22f7180d0283) quincy (dev) 1: /lib64/libpthread.so.0(+0x12b20) [0x7fd8206f5b20] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x55c6780db3a1] 5: (PrimaryLogPG::recover_replicas(unsigned long, ThreadPool::TPHandle&, bool*)+0x967) [0x55c678352b77] 6: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0x10e) [0x55c67835c9ce] 7: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x2f5) [0x55c6781de475] 8: (ceph::osd::scheduler::PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1d) [0x55c67843f8ed] 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xa58) [0x55c6781fc048] 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x55c6788874e4] 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c678888884] 12: (Thread::_entry_func(void*)+0xd) [0x55c67886e5ad] 13: /lib64/libpthread.so.0(+0x814a) [0x7fd8206eb14a] 14: clone() 2021-05-26T06:19:54.923 INFO:scrape:1 jobs: ['6135980'] 2021-05-26T06:19:54.923 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/filestore-xfs', 'rados', 'rados/singleton/{all/ec-backfill-unfound', 'supported-random-distro$/{centos_8}}'] 2021-05-26T06:19:54.924 INFO:scrape: 2021-05-26T06:19:54.925 INFO:teuthology.results:Sending results to to.my.trociny@gmail.com: Test Run: trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/ failed: 1 dead: 0 running: 0 waiting: 0 queued: 0 passed: 15 Fail ================================================================= [6135980] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 01:08:53 info: http://pulpito.ceph.com/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/6135980/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/6135980/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0fb3a4faeacc45ada4228bea7885e083 Command failed on smithi118 with status 6: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats' Pass ================================================================= [6135973] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:55:03 info: http://pulpito.ceph.com/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/6135973/ [6135974] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:23:53 info: http://pulpito.ceph.com/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/6135974/ [6135975] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:43:38 info: http://pulpito.ceph.com/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/6135975/ [6135976] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:14:22 info: http://pulpito.ceph.com/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/6135976/ [6135977] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:51:59 info: http://pulpito.ceph.com/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/6135977/ [6135978] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:45:50 info: http://pulpito.ceph.com/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/6135978/ [6135979] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:23:30 info: http://pulpito.ceph.com/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/6135979/ [6135981] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:24:03 info: http://pulpito.ceph.com/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/6135981/ [6135982] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:14:34 info: http://pulpito.ceph.com/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/6135982/ [6135983] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:24:07 info: http://pulpito.ceph.com/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/6135983/ [6135984] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:47:59 info: http://pulpito.ceph.com/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/6135984/ [6135985] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:48:28 info: http://pulpito.ceph.com/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/6135985/ [6135986] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:23:48 info: http://pulpito.ceph.com/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/6135986/ [6135987] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:15:08 info: http://pulpito.ceph.com/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/6135987/ [6135988] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:23:46 info: http://pulpito.ceph.com/trociny-2021-05-26_04:56:24-rados-master-distro-basic-smithi/6135988/