Found 262 jobs
Found 6 distinct failure reasons
Failure: SSH connection to smithi133 was lost: 'uname -r'
5 jobs: ['6629956', '6630078', '6630110', '6630084', '6629985']
suites intersection: ['rados']
suites union: ['1-install/jewel', '1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'clusters/{openstack', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'd-balancer/off', 'distro$/{ubuntu_18.04}', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/singleton/{all/backfill-toofull', 'rados/singleton/{all/mon-config-keys', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_18.04}', 'supported-random-distro$/{ubuntu_18.04}}', 'thrashers/none', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/rbd_cls}', 'workloads/redirect_set_object}']

Failure: Command failed on smithi052 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e98897febf3caebc1072d3529f386b5f75564b2f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 30de3ae2-7a43-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi052:vg_nvme/lv_4'
9 jobs: ['6630135', '6630059', '6630030', '6630189', '6630086', '6630161', '6629951', '6629980', '6630004']
suites intersection: ['ceph', 'clusters/{openstack', 'distro$/{ubuntu_18.04}', 'rados', 'thrashosds-health', 'three-plus-one}']
suites union: ['1-install/jewel-v1only', '1-install/luminous', '1-install/luminous-v1only', '1-install/mimic', '1-install/mimic-v1only', '1-install/nautilus', '1-install/nautilus-v1only', '1-install/nautilus-v2only', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/off', 'distro$/{ubuntu_18.04}', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'rados', 'rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}', 'workloads/radosbench}', 'workloads/rbd_cls}', 'workloads/snaps-few-objects}', 'workloads/test_rbd_api}']

Failure: Command failed on smithi017 with status 5: 'sudo systemctl stop ceph-c9387298-7a3a-11ec-8c35-001a4aab830c@mon.a'
2 jobs: ['6630008', '6630100']
suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2}', 'rados/cephadm/upgrade/{1-start-distro/1-start-centos_8']
suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'fixed-2}', 'rados/cephadm/upgrade/{1-start-distro/1-start-centos_8']

Failure: "2022-01-20T23:36:37.091383+0000 mon.a (mon.0) 356 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log
2 jobs: ['6630190', '6629950']
suites intersection: ['msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_8}']
suites union: ['clusters/3-mons', 'clusters/9-mons', 'msgr-failures/few', 'msgr-failures/mon-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_8}', 'thrashers/one', 'thrashers/sync-many', 'workloads/rados_mon_osdmap_prune}', 'workloads/snaps-few-objects}']

Timeout 120 running ceph --cluster ceph pg dump --format=json
1 jobs: ['6630157']
suites: ['msgr-failures/none', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/singleton/{all/rebuild-mondb', 'supported-random-distro$/{ubuntu_18.04}}']

Crash: Command failed (workunit test rados/test_crash.sh) on smithi119 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e98897febf3caebc1072d3529f386b5f75564b2f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_crash.sh'
ceph version 15.2.15-160-ge98897fe (e98897febf3caebc1072d3529f386b5f75564b2f) octopus (stable)
 1: (()+0x12980) [0x7fec01846980]
 2: (pthread_cond_wait()+0x243) [0x7fec01841ad3]
 3: (std::condition_variable::wait(std::unique_lock<std::mutex>&)+0xc) [0x7fec00f188bc]
 4: (AsyncMessenger::wait()+0x8f) [0x55cc0c335f6f]
 5: (main()+0x4050) [0x55cc0bada1e0]
 6: (__libc_start_main()+0xe7) [0x7fec004dbbf7]
 7: (_start()+0x2a) [0x55cc0baef03a]
1 jobs: ['6630180']
suites: ['msgr-failures/none', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'rados', 'rados/singleton/{all/test-crash', 'supported-random-distro$/{ubuntu_18.04}}']