Found 100 jobs Found 5 distinct failure reasons Failure: "2020-11-01T22:46:37.645166+0000 osd.5 (osd.5) 3 : cluster [WRN] slow request osd_op(client.4361.0:53584 2.3 2:cee0b40c:::rbd_data.1131508c1732.0000000000005f53:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~8192 in=8192b] snapc 0=[] RETRY=8 ondisk+retry+write+known_if_redirected e21) initiated 2020-11-01T22:46:07.255355+0000 currently waiting for sub ops" in cluster log 1 jobs: ['5577268'] suites: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'msgr-failures/many', 'tasks/rbd_simple_big}'] Failure: Command failed on smithi105 with status 120: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx --cluster ceph --id 0 -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 512 -w 512 -t 512 -h 512 -l 250000000 -S 0 -N 20000 -K -U pool_client.0 image_client.0' 1 jobs: ['5577230'] suites: ['clusters/3-node', 'conf', 'features/object-map', 'krbd/fsx/{ceph/ceph', 'objectstore/bluestore-bitmap', 'randomized-striping-off}', 'striping/default/{msgr-failures/many', 'tasks/fsx-1-client}'] Timeout 3h running clone.client.0/qa/workunits/rbd/concurrent.sh 1 jobs: ['5577253'] suites: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'msgr-failures/many', 'tasks/rbd_concurrent}'] Failure: Command failed on smithi007 with status 3: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludef8biu575 -r -- '-g auto -g blockdev -x clone'" 2 jobs: ['5577173', '5577224'] suites intersection: ['conf', 'krbd/singleton/{bluestore-bitmap', 'tasks/rbd_xfstests}'] suites union: ['conf', 'krbd/singleton/{bluestore-bitmap', 'msgr-failures/few', 'msgr-failures/many', 'tasks/rbd_xfstests}'] Failure: "2020-11-01T23:15:27.220973+0000 osd.3 (osd.3) 3 : cluster [WRN] slow request osd_op(client.81209.0:493014 2.5 2:b541cfe7:::rbd_header.114fb022a95b:head [watch reconnect cookie 18446462598732840967 gen 2877] snapc 0=[] ondisk+write+known_if_redirected e21) initiated 2020-11-01T23:14:56.882300+0000 currently started" in cluster log 1 jobs: ['5577242'] suites: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'msgr-failures/many', 'tasks/krbd_udev_enumerate}']