2021-05-26T21:14:46.784 INFO:root:teuthology version: 1.1.0-0c70841a 2021-05-26T21:14:46.786 INFO:teuthology.results:subset: '2/10' 2021-05-26T21:14:46.786 INFO:teuthology.results:seed: '7796' 2021-05-26T21:20:52.444 INFO:root:teuthology version: 1.1.0-0c70841a 2021-05-26T21:20:52.474 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-05-26T22:27:57.359 INFO:teuthology.results:Tests finished! gathering results... 2021-05-26T22:27:58.732 INFO:scrape:Found 16 jobs 2021-05-26T22:27:59.054 INFO:scrape:Found 1 distinct failure reasons 2021-05-26T22:27:59.054 INFO:scrape:Crash: Command failed on smithi115 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats' 2021-05-26T22:27:59.055 INFO:scrape:ceph version 17.0.0-4472-g5276c2a3 (5276c2a3d0eae8e5f7b7f8efd1bb22f7180d0283) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f4cc2d7d3c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1af) [0x55d2c1829321] 5: (PrimaryLogPG::recover_replicas(unsigned long, ThreadPool::TPHandle&, bool*)+0xa57) [0x55d2c1a9d8e7] 6: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0x109) [0x55d2c1aa9779] 7: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x25d) [0x55d2c18f1cdd] 8: (ceph::osd::scheduler::PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1d) [0x55d2c1ba8a3d] 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x872) [0x55d2c191e5f2] 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x403) [0x55d2c203f0a3] 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55d2c2041ec4] 12: (Thread::_entry_func(void*)+0xd) [0x55d2c203184d] 13: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f4cc2d71609] 14: clone() 2021-05-26T22:27:59.055 INFO:scrape:2 jobs: ['6136032', '6136041'] 2021-05-26T22:27:59.056 INFO:scrape:suites intersection: ['mon_election/connectivity', 'msgr/async-v2only', 'objectstore/filestore-xfs', 'rados', 'rados/singleton/{all/ec-backfill-unfound'] 2021-05-26T22:27:59.056 INFO:scrape:suites union: ['mon_election/connectivity', 'msgr-failures/many', 'msgr-failures/none', 'msgr/async-v2only', 'objectstore/filestore-xfs', 'rados', 'rados/singleton/{all/ec-backfill-unfound', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}}'] 2021-05-26T22:27:59.057 INFO:scrape: 2021-05-26T22:27:59.057 INFO:teuthology.results:Sending results to to.my.trociny@gmail.com: Test Run: trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 14 Fail ================================================================= [6136032] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:17:53 info: http://pulpito.ceph.com/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/6136032/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/6136032/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0ab8da8c00fa413e90f3a91fbdd2ed27 Command failed on smithi115 with status 6: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats' [6136041] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:28:05 info: http://pulpito.ceph.com/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/6136041/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/6136041/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c6c53c1cbce347a7bdf4b2a3878783fc Command failed on smithi002 with status 6: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats' Pass ================================================================= [6136033] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:14:23 info: http://pulpito.ceph.com/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/6136033/ [6136034] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:52:51 info: http://pulpito.ceph.com/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/6136034/ [6136035] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:54:21 info: http://pulpito.ceph.com/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/6136035/ [6136036] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:23:12 info: http://pulpito.ceph.com/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/6136036/ [6136037] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:41:34 info: http://pulpito.ceph.com/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/6136037/ [6136038] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:39:42 info: http://pulpito.ceph.com/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/6136038/ [6136039] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:22:57 info: http://pulpito.ceph.com/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/6136039/ [6136040] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:13:21 info: http://pulpito.ceph.com/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/6136040/ [6136042] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:13:52 info: http://pulpito.ceph.com/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/6136042/ [6136043] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:23:10 info: http://pulpito.ceph.com/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/6136043/ [6136044] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 01:02:46 info: http://pulpito.ceph.com/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/6136044/ [6136045] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:14:08 info: http://pulpito.ceph.com/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/6136045/ [6136046] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:23:31 info: http://pulpito.ceph.com/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/6136046/ [6136047] rados/singleton/{all/ec-backfill-unfound mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:53:15 info: http://pulpito.ceph.com/trociny-2021-05-26_06:07:06-rados-master-distro-basic-smithi/6136047/