Found 320 jobs Missing teuthology log /home/teuthworker/archive/yuriw-2023-01-11_21:29:58-rados-wip-yuri-testing-2023-01-11-0818-distro-default-smithi/7133912/teuthology.log Found 15 distinct failure reasons Failure: timeout expired in wait_until_healthy 24 jobs: ['7134017', '7134053', '7134275', '7134624', '7134509', '7133960', '7134568', '7133936', '7134610', '7134424', '7134104', '7134359', '7134218', '7134670', '7134530', '7134320', '7134035', '7134380', '7134442', '7134257', '7134152', '7133906', '7134645', '7134651'] suites intersection: [] suites union: ['0-nvme-loop', '1-start', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-services/basic', '2-services/rgw', '3-final}', 'agent/off', 'agent/on', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/osds/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun', 'rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0', 'rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8', 'rados/cephadm/osds/{0-distro/ubuntu_20.04', 'rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools}', 'rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_rhel8}', 'rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04}', 'rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun', 'rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8', 'rados/cephadm/smoke/{0-distro/ubuntu_20.04', 'start}'] Failure: {'smithi005.front.sepia.ceph.com': {'_ansible_no_log': False, 'attempts': 12, 'changed': True, 'cmd': ['subscription-manager', 'register', '--activationkey=testnode', '--org=Ceph', '--name=smithi005.front.sepia.ceph.com', '--force'], 'delta': '0:00:00.704776', 'end': '2023-01-12 02:19:56.869522', 'failed_when_result': True, 'invocation': {'module_args': {'_raw_params': 'subscription-manager register --activationkey=testnode --org=Ceph --name=smithi005.front.sepia.ceph.com --force', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 70, 'start': '2023-01-12 02:19:56.164746', 'stderr': 'The DMI UUID of this host (00000000-0000-0000-0000-0CC47A6BFDDE) matches other registered hosts: smithi005 (HTTP error code 422: Unprocessable Entity)', 'stderr_lines': ['The DMI UUID of this host (00000000-0000-0000-0000-0CC47A6BFDDE) matches other registered hosts: smithi005 (HTTP error code 422: Unprocessable Entity)'], 'stdout': '', 'stdout_lines': []}} 24 jobs: ['7134044', '7134077', '7134657', '7134389', '7134582', '7134197', '7133915', '7134134', '7134050', '7134590', '7134546', '7134305', '7133894', '7134341', '7134588', '7134488', '7134642', '7134532', '7134383', '7134608', '7134022', '7134314', '7134338', '7134647'] suites intersection: [] suites union: ['0-nvme-loop', '1-pg-log-overrides/short_pg_log', '1-start', '2-ops/repave-all}', '2-ops/rm-zap-flag}', '2-ops/rmdir-reactivate}', '2-recovery-overrides/{more-async-recovery}', '2-services/basic', '3-final}', '3-scrub-overrides/{default}', 'agent/off', 'backoff/peering_and_degraded', 'ceph', 'cluster/{12-osds', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'debug/mgr', 'fast/fast', 'fixed-2', 'mgr_ttl_cache/disable', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0', 'rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8', 'rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_rhel8}', 'rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8', 'rados/mgr/{clusters/{2-node-mgr}', 'rados/singleton-nomsgr/{all/cache-fs-trunc', 'rados/singleton-nomsgr/{all/export-after-evict', 'rados/singleton-nomsgr/{all/full-tiering', 'rados/singleton-nomsgr/{all/msgr', 'rados/singleton-nomsgr/{all/pool-access', 'rados/singleton/{all/recovery-preemption', 'rados/standalone/{supported-random-distro$/{rhel_8}', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'random-objectstore$/{bluestore-comp-zstd}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'start}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'tasks/libcephsqlite}', 'tasks/progress}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rados_workunit_loadgen_mix}', 'tasks/repair_test}', 'thrashers/careful', 'thrashers/default', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}', 'workloads/osd-backfill}', 'workloads/radosbench}'] Failure: Test failure: test_non_existent_cluster (tasks.cephfs.test_nfs.TestNFS) 2 jobs: ['7134558', '7134170'] suites intersection: ['task/test_nfs}'] suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_nfs}'] Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi114 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50fd2994f91565b07a1fd77e4f8e8a03dc49f8a7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2 jobs: ['7134221', '7134638'] suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'tasks/e2e}'] suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-lz4}', 'random-objectstore$/{bluestore-comp-snappy}', 'tasks/e2e}'] Failure: Test failure: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI) 1 jobs: ['7134596'] suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_orch_cli}'] Failure: Command failed (workunit test post-file.sh) on smithi033 with status 255: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50fd2994f91565b07a1fd77e4f8e8a03dc49f8a7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/post-file.sh' 2 jobs: ['7134560', '7134089'] suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-post-file'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-post-file', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}'] Failure: SELinux denials found on ubuntu@smithi012.front.sepia.ceph.com: ['type=AVC msg=audit(1673508240.832:19408): avc: denied { ioctl } for pid=122489 comm="iptables" path="/var/lib/containers/storage/overlay/35f20ebafcaea7e0ee7e8262615826256f0aaa2504a0b6a7fd8297ed1ba00590/merged" dev="overlay" ino=3412250 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1'] 1 jobs: ['7134397'] suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm}'] Failure: Command failed on smithi040 with status 1: 'sudo yum -y install ceph-test' 6 jobs: ['7133978', '7134542', '7134457', '7134640', '7134224', '7133984'] suites intersection: [] suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', 'agent/on', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'mon_election/connectivity', 'net/calico', 'net/flannel', 'net/host', 'rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'rados/objectstore/{backends/alloc-hint', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}', 'rook/master}', 'supported-random-distro$/{rhel_8}}', 'task/test_iscsi_pids_limit/{centos_8.stream_container_tools', 'test_iscsi_pids_limit}}'] Failure: SELinux denials found on ubuntu@smithi046.front.sepia.ceph.com: ['type=AVC msg=audit(1673486964.813:19245): avc: denied { ioctl } for pid=106991 comm="iptables" path="/var/lib/containers/storage/overlay/2c3fe7a2558ea55a0c07ffc39e91bce086df7aec47434330ebd61e1c0e0c06fd/merged" dev="overlay" ino=3540940 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1', 'type=AVC msg=audit(1673486964.865:19246): avc: denied { ioctl } for pid=106995 comm="iptables" path="/var/lib/containers/storage/overlay/2c3fe7a2558ea55a0c07ffc39e91bce086df7aec47434330ebd61e1c0e0c06fd/merged" dev="overlay" ino=3540940 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1'] 1 jobs: ['7133998'] suites: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_cephadm}'] Failure: Command failed (workunit test rados/test_envlibrados_for_rocksdb.sh) on smithi169 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50fd2994f91565b07a1fd77e4f8e8a03dc49f8a7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_envlibrados_for_rocksdb.sh' 1 jobs: ['7134173'] suites: ['mon_election/connectivity', 'msgr-failures/none', 'msgr/async-v2only', 'objectstore/filestore-xfs', 'rados', 'rados/singleton/{all/test_envlibrados_for_rocksdb', 'supported-random-distro$/{rhel_8}}'] Failure: Test failure: test_prometheus (tasks.mgr.test_module_selftest.TestModuleSelftest) 1 jobs: ['7134293'] suites: ['debug/mgr', 'mgr_ttl_cache/enable', 'mon_election/connectivity', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore-comp-zlib}', 'supported-random-distro$/{centos_8}', 'tasks/module_selftest}'] Failure: ceph version 17.2.5-654.g13e3f6e2 was not installed, found 17.2.5-652.g16e8d248.el8. 1 jobs: ['7134506'] suites: ['0-size-min-size-overrides/3-size-2-min-size', '1-install/quincy', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'rados', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'thrashers/morepggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/snaps-few-objects}'] Crash: Command failed (workunit test rados/test_crash.sh) on smithi182 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50fd2994f91565b07a1fd77e4f8e8a03dc49f8a7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_crash.sh' ceph version 18.0.0-1763-g50fd2994 (50fd2994f91565b07a1fd77e4f8e8a03dc49f8a7) reef (dev) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f404d1b6cf0] 2: pthread_cond_wait() 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f404c7f98f0] 4: (AsyncMessenger::wait()+0x147) [0x558cdaf2d107] 5: main() 6: __libc_start_main() 7: _start() 1 jobs: ['7134128'] suites: ['mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/singleton/{all/test-crash', 'supported-random-distro$/{centos_8}}'] Dead: None 1 jobs: ['7133912'] Failure: Command failed on smithi006 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to-stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug-bluestore 20" ceph_test_objectstore --gtest_filter=*/2:-*SyntheticMatrixC* --gtest_catch_exceptions=0\'' 1 jobs: ['7134630'] suites: ['rados/objectstore/{backends/objectstore-bluestore-a', 'supported-random-distro$/{ubuntu_latest}}']