2020-09-20T17:32:31.146 INFO:root:teuthology version: 1.0.0-f390fd3 2020-09-20T17:32:31.150 INFO:teuthology.results:subset: '888/9999' 2020-09-20T17:32:31.150 INFO:teuthology.results:seed: '2477' 2020-09-20T17:36:14.936 INFO:root:teuthology version: 1.0.0-f390fd3 2020-09-20T17:36:14.958 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-09-20T18:08:15.522 INFO:teuthology.results:Tests finished! gathering results... 2020-09-20T18:08:15.869 INFO:scrape:Found 3 jobs 2020-09-20T18:08:16.323 INFO:scrape:Found 2 distinct failure reasons 2020-09-20T18:08:16.324 INFO:scrape:Crash: Command failed on smithi023 with status 6: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph tell osd.1 injectargs -- --filestore_debug_random_read_err=0.0'" 2020-09-20T18:08:16.325 INFO:scrape:ceph version 16.0.0-5625-g1d421102 (1d421102af66c91b42b4c769c7a88260d1a0efc3) pacific (dev) 1: () 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x56061ea61c58] 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x56061ea61de2] 6: (ReplicatedBackend::prepare_pull(eversion_t, hobject_t const&, std::shared_ptr, ReplicatedBackend::RPGHandle*)+0x2cc) [0x56061ee485bc] 7: (ReplicatedBackend::recover_object(hobject_t const&, eversion_t, std::shared_ptr, std::shared_ptr, PGBackend::RecoveryHandle*)+0xd8) [0x56061ee55f78] 8: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0x1d1) [0x56061ec11a81] 9: (PrimaryLogPG::recover_primary(unsigned long, ThreadPool::TPHandle&)+0xb10) [0x56061ec65520] 10: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0x260) [0x56061ec70f00] 11: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x249) [0x56061eae5db9] 12: (ceph::osd::scheduler::PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x19) [0x56061ed510e9] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90c) [0x56061eb03fac] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x56061f14de2c] 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x56061f1510e0] 16: () 17: clone() 2020-09-20T18:08:16.325 INFO:scrape:2 jobs: ['5452429', '5452431'] 2020-09-20T18:08:16.325 INFO:scrape:suites intersection: ['msgr-failures/many', 'rados', 'rados/singleton/{all/random-eio', 'supported-random-distro$/{ubuntu_latest}}'] 2020-09-20T18:08:16.325 INFO:scrape:suites union: ['msgr-failures/many', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'rados', 'rados/singleton/{all/random-eio', 'supported-random-distro$/{ubuntu_latest}}'] 2020-09-20T18:08:16.326 INFO:scrape: 2020-09-20T18:08:16.326 INFO:scrape:Failure: Command failed (workunit test mon/mon-handle-forward.sh) on smithi081 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1d421102af66c91b42b4c769c7a88260d1a0efc3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-handle-forward.sh' 2020-09-20T18:08:16.326 INFO:scrape:1 jobs: ['5452430'] 2020-09-20T18:08:16.327 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/mon}'] 2020-09-20T18:08:16.327 INFO:scrape: 2020-09-20T18:08:16.327 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2020-09-20_17:31:42-rados-wip-kefu-testing-2020-09-18-1518-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2020-09-20_17:31:42-rados-wip-kefu-testing-2020-09-18-1518-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2020-09-20_17:31:42-rados-wip-kefu-testing-2020-09-18-1518-distro-basic-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [5452429] rados/singleton/{all/random-eio msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:24:50 info: http://pulpito.ceph.com/kchai-2020-09-20_17:31:42-rados-wip-kefu-testing-2020-09-18-1518-distro-basic-smithi/5452429/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-09-20_17:31:42-rados-wip-kefu-testing-2020-09-18-1518-distro-basic-smithi/5452429/ sentry: https://sentry.ceph.com/sentry/sepia/?query=aa32844f63d5483bb9962d5e8bdb2e8c Command failed on smithi023 with status 6: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph tell osd.1 injectargs -- --filestore_debug_random_read_err=0.0'" [5452430] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon} ----------------------------------------------------------------- time: 00:09:11 info: http://pulpito.ceph.com/kchai-2020-09-20_17:31:42-rados-wip-kefu-testing-2020-09-18-1518-distro-basic-smithi/5452430/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-09-20_17:31:42-rados-wip-kefu-testing-2020-09-18-1518-distro-basic-smithi/5452430/ sentry: https://sentry.ceph.com/sentry/sepia/?query=896c6d2e25fe44b88a416634d6c8e0d5 Command failed (workunit test mon/mon-handle-forward.sh) on smithi081 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1d421102af66c91b42b4c769c7a88260d1a0efc3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-handle- forward.sh' [5452431] rados/singleton/{all/random-eio msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:23:44 info: http://pulpito.ceph.com/kchai-2020-09-20_17:31:42-rados-wip-kefu-testing-2020-09-18-1518-distro-basic-smithi/5452431/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-09-20_17:31:42-rados-wip-kefu-testing-2020-09-18-1518-distro-basic-smithi/5452431/ sentry: https://sentry.ceph.com/sentry/sepia/?query=a87d7393637c43f3ad0c6d05290e6849 Command failed on smithi171 with status 6: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph tell osd.1 injectargs -- --filestore_debug_random_read_err=0.0'"