Found 515 jobs Found 7 distinct failure reasons Failure: Command failed (workunit test cephtool/test.sh) on smithi098 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=74f48adff35db6f86e9231614da019ef946277a3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 1 jobs: ['5738475'] suites: ['msgr-failures/many', 'msgr/async', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_latest}}'] Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2 jobs: ['5738618', '5738709'] suites intersection: ['1-pg-log-overrides/normal_pg_log', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/upmap', 'msgr-failures/osd-delay', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashosds-health', 'workloads/rados_api_tests}'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/upmap', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/careful', 'thrashers/default', 'thrashosds-health', 'workloads/rados_api_tests}'] Failure: SSH connection to smithi001 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' 1 jobs: ['5738583'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'msgr-failures/fastclose', 'msgr/random', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_16.04}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-pool-snaps-readproxy}'] Failure: Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress) 1 jobs: ['5738700'] suites: ['debug/mgr', 'objectstore/bluestore-comp-zstd', 'rados/mgr/{clusters/{2-node-mgr}', 'supported-random-distro$/{rhel_7}', 'tasks/progress}'] Failure: "2020-12-31 23:33:20.178594 mon.a (mon.0) 105 : cluster [WRN] Health check failed: Degraded data redundancy: 174/19446 objects degraded (0.895%), 2 pgs degraded (PG_DEGRADED)" in cluster log 1 jobs: ['5738424'] suites: ['rados', 'rados/singleton-nomsgr/{all/balancer', 'supported-random-distro$/{rhel_7}}'] Failure: SSH connection to smithi154 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' 1 jobs: ['5738808'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'msgr-failures/fastclose', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_7}', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}'] Crash: Command failed on smithi192 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.2 flush_pg_stats' ceph version 14.2.16-114-g74f48adff3 (74f48adff35db6f86e9231614da019ef946277a3) nautilus (stable) 1: (()+0xf630) [0x7fa87408c630] 2: (gsignal()+0x37) [0x7fa872e80387] 3: (abort()+0x148) [0x7fa872e81a78] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x56105c548e78] 5: (()+0x4d8ff1) [0x56105c548ff1] 6: (bool PGLog::append_log_entries_update_missing >(hobject_t const&, bool, std::list > const&, bool, PGLog::IndexedLog*, pg_missing_set&, PGLog::LogEntryHandler*, DoutPrefixProvider const*)+0x5cd) [0x56105c766ffd] 7: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0xdea) [0x56105c75c26a] 8: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x64) [0x56105c6be894] 9: (PG::RecoveryState::Stray::react(MLogRec const&)+0x22b) [0x56105c70144b] 10: (boost::statechart::simple_state, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xa5) [0x56105c74fa55] 11: (PG::do_peering_event(std::shared_ptr, PG::RecoveryCtx*)+0x2dd) [0x56105c713a4d] 12: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr, ThreadPool::TPHandle&)+0x1b4) [0x56105c6505f4] 13: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x51) [0x56105c8b88f1] 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x56105c64511f] 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x56105cbfd8d6] 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x56105cc003f0] 17: (()+0x7ea5) [0x7fa874084ea5] 18: (clone()+0x6d) [0x7fa872f488dd] 1 jobs: ['5738526'] suites: ['msgr-failures/few', 'msgr/random', 'objectstore/bluestore-comp-zstd', 'rados', 'rados/singleton/{all/osd-recovery', 'supported-random-distro$/{centos_latest}}']