2020-09-23T14:46:43.985 INFO:root:teuthology version: 1.0.0-f390fd3 2020-09-23T14:46:43.986 INFO:teuthology.results:subset: '888/9999' 2020-09-23T14:46:43.986 INFO:teuthology.results:seed: '2477' 2020-09-23T14:50:45.575 INFO:root:teuthology version: 1.0.0-f390fd3 2020-09-23T14:50:45.746 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-09-23T15:25:46.362 INFO:teuthology.results:Tests finished! gathering results... 2020-09-23T15:25:46.632 INFO:scrape:Found 3 jobs 2020-09-23T15:25:47.136 INFO:scrape:Found 2 distinct failure reasons 2020-09-23T15:25:47.138 INFO:scrape:Crash: Command failed on smithi062 with status 6: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph tell osd.1 injectargs -- --filestore_debug_random_read_err=0.0'" 2020-09-23T15:25:47.138 INFO:scrape:ceph version 16.0.0-5761-g86638684 (86638684894f4489730258515a961631752ba196) pacific (dev) 1: () 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)ve_ptr&, ThreadPool::TPHandle&)kend14RecoveryHandleE+0x19c) [0x55cbdb6420b8] 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)Pool::TPHandle&)kend14RecoveryHandleE+0) [0x55cbdb642242] 6: (ReplicatedBackend::prepare_pull(eversion_t, hobject_t const&, std::shared_ptr, ReplicatedBackend::RPGHandle*)E+0x2cc) [0x55cbdba2ca3c] 7: (ReplicatedBackend::recover_object(hobject_t const&, eversion_t, std::shared_ptr, std::shared_ptr, PGBackend::RecoveryHandle*)P�`_���U`_���U� ��U+0xd8) [0x55cbdba3a3f8] 8: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)ES6_PN9PGBackend14RecoveryHandleE+0x1d1) [0x55cbdb7f1f41] 9: (PrimaryLogPG::recover_primary(unsigned long, ThreadPool::TPHandle&)ackend::RecoveryHandle*)ES6_PN9PGBackend14RecoveryHandleE+0xb10) [0x55cbdb845a80] 10: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)dle*)ES6_PN9PGBackend14RecoveryHandleE+0x260) [0x55cbdb851340] 11: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)igned long*)dle*)ES6_PN9PGBackend14RecoveryHandleE+0x249) [0x55cbdb6c62b9] 12: (ceph::osd::scheduler::PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)kend14RecoveryHandleE+0x19) [0x55cbdb9322e9] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)ive_ptr&, ThreadPool::TPHandle&)kend14RecoveryHandleE+0x90c) [0x55cbdb6e44ac] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)E+0x4ac) [0x55cbdbd322ac] 15: (ShardedThreadPool::WorkThreadSharded::entry()signed int)E+0x10) [0x55cbdbd35560] 16: () 17: clone() 2020-09-23T15:25:47.139 INFO:scrape:2 jobs: ['5462883', '5462881'] 2020-09-23T15:25:47.139 INFO:scrape:suites intersection: ['msgr-failures/many', 'rados', 'rados/singleton/{all/random-eio', 'supported-random-distro$/{ubuntu_latest}}'] 2020-09-23T15:25:47.140 INFO:scrape:suites union: ['msgr-failures/many', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'rados', 'rados/singleton/{all/random-eio', 'supported-random-distro$/{ubuntu_latest}}'] 2020-09-23T15:25:47.140 INFO:scrape: 2020-09-23T15:25:47.141 INFO:scrape:Failure: Command failed (workunit test mon/misc.sh) on smithi045 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1d421102af66c91b42b4c769c7a88260d1a0efc3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/misc.sh' 2020-09-23T15:25:47.141 INFO:scrape:1 jobs: ['5462882'] 2020-09-23T15:25:47.141 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/mon}'] 2020-09-23T15:25:47.142 INFO:scrape: 2020-09-23T15:25:47.142 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2020-09-23_14:44:22-rados-wip-kefu2-testing-2020-09-21-1826-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2020-09-23_14:44:22-rados-wip-kefu2-testing-2020-09-21-1826-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2020-09-23_14:44:22-rados-wip-kefu2-testing-2020-09-21-1826-distro-basic-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [5462881] rados/singleton/{all/random-eio msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:25:34 info: http://pulpito.ceph.com/kchai-2020-09-23_14:44:22-rados-wip-kefu2-testing-2020-09-21-1826-distro-basic-smithi/5462881/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-09-23_14:44:22-rados-wip-kefu2-testing-2020-09-21-1826-distro-basic-smithi/5462881/ sentry: https://sentry.ceph.com/sentry/sepia/?query=b6f2ecb065e143a98a9f4e91bdf2c3cb Command failed on smithi205 with status 6: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph tell osd.1 injectargs -- --filestore_debug_random_read_err=0.0'" [5462882] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon} ----------------------------------------------------------------- time: 00:08:19 info: http://pulpito.ceph.com/kchai-2020-09-23_14:44:22-rados-wip-kefu2-testing-2020-09-21-1826-distro-basic-smithi/5462882/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-09-23_14:44:22-rados-wip-kefu2-testing-2020-09-21-1826-distro-basic-smithi/5462882/ sentry: https://sentry.ceph.com/sentry/sepia/?query=517174c4fdef47979a2c63e839d881d9 Command failed (workunit test mon/misc.sh) on smithi045 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1d421102af66c91b42b4c769c7a88260d1a0efc3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/misc.sh' [5462883] rados/singleton/{all/random-eio msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:24:10 info: http://pulpito.ceph.com/kchai-2020-09-23_14:44:22-rados-wip-kefu2-testing-2020-09-21-1826-distro-basic-smithi/5462883/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-09-23_14:44:22-rados-wip-kefu2-testing-2020-09-21-1826-distro-basic-smithi/5462883/ sentry: https://sentry.ceph.com/sentry/sepia/?query=aad5ab1d2461444087db024da992a182 Command failed on smithi062 with status 6: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph tell osd.1 injectargs -- --filestore_debug_random_read_err=0.0'"