Found 151 jobs Found 2 distinct failure reasons Failure: "2020-10-19T21:31:32.600763+0000 osd.0 (osd.0) 3 : cluster [WRN] slow request osd_op(client.10020.0:4103 3.3 3:e1676c44:::rbd_data.27218db0d5cd.0000000000000046:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3497984~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected e113) initiated 2020-10-19T21:31:01.842377+0000 currently reached pg" in cluster log 1 jobs: ['5531849'] suites: ['clusters/{fixed-1', 'features/layering', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'pool/small-cache-pool', 'rbd/cli/{base/install', 'supported-random-distro$/{centos_latest}', 'workloads/rbd_cli_generic}'] Crash: Command failed (workunit test rbd/rbd_mirror_fsx_compare.sh) on smithi036 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2f33da11da69822d9bab6e9cadb58a7cc3d5720f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ARGS=\'\' RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_fsx_compare.sh' ceph version 15.2.5-355-g2f33da11 (2f33da11da69822d9bab6e9cadb58a7cc3d5720f) octopus (stable) 1: (()+0x12dc0) [0x7ff802a5bdc0] 2: (gsignal()+0x10f) [0x7ff800efb8df] 3: (abort()+0x127) [0x7ff800ee5cf5] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7ff80385d605] 5: (()+0x27a7ce) [0x7ff80385d7ce] 6: (librbd::image::CloseRequest::handle_shut_down_exclusive_lock(int)+0x617) [0x55e942a7dc47] 7: (Context::complete(int)+0xd) [0x55e9427b09bd] 8: (Context::complete(int)+0xd) [0x55e9427b09bd] 9: (librbd::ManagedLock::complete_shutdown(int)+0x195) [0x55e942991825] 10: (Context::complete(int)+0xd) [0x55e9427b09bd] 11: (librbd::ManagedLock::wait_for_tracked_ops(int)+0x22b) [0x55e94298effb] 12: (librbd::ManagedLock::handle_shutdown(int)+0x204) [0x55e94298f2c4] 13: (ThreadPool::PointerWQ::_void_process(void*, ThreadPool::TPHandle&)+0x148) [0x55e9427d4a88] 14: (ThreadPool::worker(ThreadPool::WorkThread*)+0xe64) [0x7ff803949d44] 15: (ThreadPool::WorkThread::entry()+0x15) [0x7ff80394a5a5] 16: (()+0x82de) [0x7ff802a512de] 17: (clone()+0x43) [0x7ff800fc0133] 1 jobs: ['5531949'] suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_latest}', 'workloads/rbd-mirror-fsx-workunit}']