Found 20 jobs
Found 8 distinct failure reasons
Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi116 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8a074fcf8834e0bf886be04d53b6ac80280d9574 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
3 jobs: ['6666473', '6666479', '6666466']
suites intersection: ['task/test_cephadm}']
suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{agent/off', 'rados/cephadm/workunits/{agent/on', 'task/test_cephadm}']

Failure: reached maximum tries (800) after waiting for 4800 seconds
1 jobs: ['6666471']
suites: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{default}', 'supported-random-distro$/{rhel_8}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-radosbench}']

Dead: b'2022-02-07T21:23:03.201 DEBUG:teuthology.exit:Finished running handlers'
2 jobs: ['6666470', '6666477']
suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/valgrind}']
suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/valgrind}']

Failure: "2022-02-07T15:08:18.691357+0000 osd.6 (osd.6) 43 : cluster [ERR] 1.7 shard 6 soid 1:e00d30ae:::benchmark_data_smithi068_24814_object255:head : " in cluster log
1 jobs: ['6666480']
suites: ['clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/many', 'msgr/async', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'tasks/scrub_test}']

Failure: Command failed on smithi062 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:8a074fcf8834e0bf886be04d53b6ac80280d9574 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f55768e0-8825-11ec-8c35-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch ps\nHOST=$(hostname -s)\nOSD=$(ceph orch ps $HOST | grep osd | head -n 1 | awk \'"\'"\'{print $1}\'"\'"\')\necho "host $HOST, osd $OSD"\nceph orch daemon stop $OSD\nwhile ceph orch ps | grep $OSD | grep running ; do sleep 5 ; done\nceph auth export $OSD > k\nceph orch daemon rm $OSD --force\nceph orch ps --refresh\nwhile ceph orch ps | grep $OSD ; do sleep 5 ; done\nceph auth add $OSD -i k\nceph cephadm osd activate $HOST\nwhile ! ceph orch ps | grep $OSD | grep running ; do sleep 5 ; done\n\''
3 jobs: ['6666468', '6666481', '6666475']
suites intersection: ['0-nvme-loop', '1-start', '2-ops/rmdir-reactivate}']
suites union: ['0-nvme-loop', '1-start', '2-ops/rmdir-reactivate}', 'rados/cephadm/osds/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8', 'rados/cephadm/osds/{0-distro/ubuntu_20.04']

Crash: Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi026 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8a074fcf8834e0bf886be04d53b6ac80280d9574 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh'
ceph version 17.0.0-10572-g8a074fcf (8a074fcf8834e0bf886be04d53b6ac80280d9574) quincy (dev)
 1: /lib64/libpthread.so.0(+0x12c20) [0x682dc20]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x6e0b78]
 5: ceph-osd(+0x5d8d3b) [0x6e0d3b]
 6: ceph-osd(+0xdfd977) [0xf05977]
 7: (MgrClient::~MgrClient()+0x117) [0x87df27]
 8: (OSD::~OSD()+0x6ec) [0x82962c]
 9: (OSD::~OSD()+0xd) [0x82973d]
 10: main()
 11: __libc_start_main()
 12: _start()
2 jobs: ['6666484', '6666472']
suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'thrashosds-health}', 'validater/valgrind}']
suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_cls_all', 'thrashosds-health}', 'validater/valgrind}']

Failure: Command failed on smithi074 with status 251: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
4 jobs: ['6666474', '6666478', '6666482', '6666469']
suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', '3-final', 'k8s/1.21', 'rados/rook/smoke/{0-distro/ubuntu_20.04']
suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', '3-final', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'net/flannel', 'net/host', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}', 'rook/master}']

Failure: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi084 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8a074fcf8834e0bf886be04d53b6ac80280d9574 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh'
1 jobs: ['6666467']
suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/scrub}']