2022-12-09T06:44:31.370 INFO:root:teuthology version: 0.0.1.dev18+g19d18db
2022-12-09T06:44:31.370 INFO:teuthology.results:subset: '111/120000'
2022-12-09T06:44:31.370 INFO:teuthology.results:seed: '4612'
2022-12-09T06:57:02.981 INFO:root:teuthology version: 0.0.1.dev18+g19d18db
2022-12-09T06:57:03.015 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-12-09T18:58:01.724 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2022-12-09T18:58:03.196 INFO:scrape:Found 18 jobs
2022-12-09T18:58:05.384 INFO:scrape:Found 8 distinct failure reasons
2022-12-09T18:58:05.385 INFO:scrape:Failure: Command failed (workunit test post-file.sh) on smithi203 with status 255: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0a1e06a0fe68e93ac914382bd642ac7088160ca7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/post-file.sh'
2022-12-09T18:58:05.385 INFO:scrape:2 jobs: ['7108937', '7108925']
2022-12-09T18:58:05.385 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-post-file', 'supported-random-distro$/{ubuntu_latest}}']
2022-12-09T18:58:05.386 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-post-file', 'supported-random-distro$/{ubuntu_latest}}']
2022-12-09T18:58:05.386 INFO:scrape:
2022-12-09T18:58:05.386 INFO:scrape:Failure: 'check osd count' reached maximum tries (90) after waiting for 900 seconds
2022-12-09T18:58:05.387 INFO:scrape:2 jobs: ['7108928', '7108940']
2022-12-09T18:58:05.387 INFO:scrape:suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', 'cluster/3-node', 'k8s/1.21', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/master}']
2022-12-09T18:58:05.387 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'net/flannel', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/master}']
2022-12-09T18:58:05.388 INFO:scrape:
2022-12-09T18:58:05.388 INFO:scrape:Failure: Command failed on smithi181 with status 1: 'TESTDIR=/home/ubuntu/cephtest bash -s'
2022-12-09T18:58:05.389 INFO:scrape:1 jobs: ['7108935']
2022-12-09T18:58:05.389 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'task/test_iscsi_pids_limit/{centos_8.stream_container_tools', 'test_iscsi_pids_limit}}']
2022-12-09T18:58:05.389 INFO:scrape:
2022-12-09T18:58:05.390 INFO:scrape:Failure: Command failed on smithi165 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to-stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug-bluestore 20" ceph_test_objectstore --gtest_filter=*/2:-*SyntheticMatrixC* --gtest_catch_exceptions=0\''
2022-12-09T18:58:05.390 INFO:scrape:1 jobs: ['7108938']
2022-12-09T18:58:05.390 INFO:scrape:suites: ['rados/objectstore/{backends/objectstore-bluestore-a', 'supported-random-distro$/{centos_8}}']
2022-12-09T18:58:05.391 INFO:scrape:
2022-12-09T18:58:05.391 INFO:scrape:Failure: Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS)
2022-12-09T18:58:05.392 INFO:scrape:2 jobs: ['7108927', '7108936']
2022-12-09T18:58:05.392 INFO:scrape:suites intersection: ['task/test_nfs}']
2022-12-09T18:58:05.392 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_nfs}']
2022-12-09T18:58:05.392 INFO:scrape:
2022-12-09T18:58:05.393 INFO:scrape:Crash: Command failed (workunit test rados/test_crash.sh) on smithi032 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0a1e06a0fe68e93ac914382bd642ac7088160ca7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_crash.sh'
2022-12-09T18:58:05.393 INFO:scrape:ceph version 18.0.0-1241-g0a1e06a0 (0a1e06a0fe68e93ac914382bd642ac7088160ca7) reef (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fdfcdd68420]
 2: pthread_cond_wait()
 3: (std::condition_variable::wait(std::unique_lock<std::mutex>&)+0x10) [0x55bf978d7f00]
 4: (AsyncMessenger::wait()+0x147) [0x55bf970061d7]
 5: main()
 6: __libc_start_main()
 7: _start()
2022-12-09T18:58:05.393 INFO:scrape:1 jobs: ['7108926']
2022-12-09T18:58:05.394 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/singleton/{all/test-crash', 'supported-random-distro$/{ubuntu_latest}}']
2022-12-09T18:58:05.394 INFO:scrape:
2022-12-09T18:58:05.394 INFO:scrape:Dead: b'2022-12-09T18:58:04.987 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight'
2022-12-09T18:58:05.395 INFO:scrape:1 jobs: ['7108939']
2022-12-09T18:58:05.395 INFO:scrape:suites: ['clusters/{fixed-2', 'fast/fast', 'mon_election/classic', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{rhel_8}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=clay-k=4-m=2}']
2022-12-09T18:58:05.396 INFO:scrape:
2022-12-09T18:58:05.396 INFO:scrape:Failure: Command failed on smithi143 with status 1: 'kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml'
2022-12-09T18:58:05.396 INFO:scrape:2 jobs: ['7108924', '7108934']
2022-12-09T18:58:05.397 INFO:scrape:suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/radosbench', 'cluster/1-node', 'k8s/1.21', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}']
2022-12-09T18:58:05.397 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/radosbench', 'cluster/1-node', 'k8s/1.21', 'net/calico', 'net/host', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}']
2022-12-09T18:58:05.397 INFO:scrape: