2022-01-20T12:25:35.364 INFO:root:teuthology version: 1.1.0-75f1bcd2 2022-01-20T12:25:35.366 INFO:teuthology.results:subset: '1/80000' 2022-01-20T12:25:35.366 INFO:teuthology.results:seed: '4174' 2022-01-20T13:58:45.263 INFO:root:teuthology version: 1.1.0-75f1bcd2 2022-01-20T13:58:45.301 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-01-20T20:35:18.658 INFO:teuthology.results:Tests finished! gathering results... 2022-01-20T20:35:35.375 INFO:scrape:Found 180 jobs 2022-01-20T20:35:38.048 INFO:scrape:Found 8 distinct failure reasons 2022-01-20T20:35:38.049 INFO:scrape:Dead: b'2022-01-20T19:58:08.173 DEBUG:teuthology.exit:Finished running handlers' 2022-01-20T20:35:38.049 INFO:scrape:7 jobs: ['6624029', '6624095', '6623953', '6623976', '6623948', '6624048', '6624022'] 2022-01-20T20:35:38.050 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '3-final}'] 2022-01-20T20:35:38.050 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs-ingress', '2-services/nfs-ingress2', '2-services/rgw-ingress', '3-final}', 'rados/cephadm/smoke-roleless/{0-distro/centos_8.2_container_tools_3.0', 'rados/cephadm/smoke-roleless/{0-distro/centos_8.3_container_tools_3.0', 'rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0', 'rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8', 'rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04'] 2022-01-20T20:35:38.051 INFO:scrape: 2022-01-20T20:35:38.051 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi163 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=843d2f8af50364375d4ff33aec4d61a68ebbfb41 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2022-01-20T20:35:38.052 INFO:scrape:1 jobs: ['6624060'] 2022-01-20T20:35:38.052 INFO:scrape:suites: ['debug/mgr', 'mon_election/classic', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-low-osd-mem-target}', 'tasks/e2e}'] 2022-01-20T20:35:38.053 INFO:scrape: 2022-01-20T20:35:38.054 INFO:scrape:Timeout 3h running clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2022-01-20T20:35:38.054 INFO:scrape:1 jobs: ['6624084'] 2022-01-20T20:35:38.054 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{rhel_8}', 'workloads/scrub}'] 2022-01-20T20:35:38.055 INFO:scrape: 2022-01-20T20:35:38.055 INFO:scrape:Failure: Command failed on smithi042 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:843d2f8af50364375d4ff33aec4d61a68ebbfb41 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c67f742-79f3-11ec-8c35-001a4aab830c -- ceph mon dump -f json' 2022-01-20T20:35:38.056 INFO:scrape:2 jobs: ['6623984', '6624010'] 2022-01-20T20:35:38.056 INFO:scrape:suites intersection: ['1-start'] 2022-01-20T20:35:38.057 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs-ingress-rgw-bucket', '2-thrash', '3-final}', '3-tasks/radosbench', 'fixed-2', 'msgr/async-v1only', 'rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8', 'rados/cephadm/thrash/{0-distro/centos_8.2_container_tools_3.0', 'root}'] 2022-01-20T20:35:38.057 INFO:scrape: 2022-01-20T20:35:38.058 INFO:scrape:Failure: Command failed (workunit test erasure-code/test-erasure-code.sh) on smithi041 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=843d2f8af50364375d4ff33aec4d61a68ebbfb41 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh' 2022-01-20T20:35:38.058 INFO:scrape:1 jobs: ['6623971'] 2022-01-20T20:35:38.059 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/erasure-code}'] 2022-01-20T20:35:38.059 INFO:scrape: 2022-01-20T20:35:38.060 INFO:scrape:Failure: Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) 2022-01-20T20:35:38.060 INFO:scrape:1 jobs: ['6624061'] 2022-01-20T20:35:38.061 INFO:scrape:suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{agent/on', 'task/test_nfs}'] 2022-01-20T20:35:38.061 INFO:scrape: 2022-01-20T20:35:38.061 INFO:scrape:Crash: "2022-01-20T14:12:55.614558+0000 mon.a (mon.0) 260 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log 2022-01-20T20:35:38.062 INFO:scrape:ceph version 17.0.0-10093-g843d2f8a (843d2f8af50364375d4ff33aec4d61a68ebbfb41) quincy (dev) 1: /lib64/libpthread.so.0(+0x12b20) [0x6825b20] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x6da682] 5: (FileStore::_do_transaction(ceph::os::Transaction&, unsigned long, int, ThreadPool::TPHandle*, char const*)+0x3f2c) [0xc6892c] 6: (FileStore::_do_transactions(std::vector >&, unsigned long, ThreadPool::TPHandle*, char const*)+0x50) [0xc6b500] 7: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x5a7) [0xc6bae7] 8: (ThreadPool::worker(ThreadPool::WorkThread*)+0xc2d) [0xeff78d] 9: (ThreadPool::WorkThread::entry()+0x15) [0xf01e15] 10: /lib64/libpthread.so.0(+0x814a) [0x681b14a] 11: clone() 2022-01-20T20:35:38.062 INFO:scrape:1 jobs: ['6624050'] 2022-01-20T20:35:38.063 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] 2022-01-20T20:35:38.063 INFO:scrape: 2022-01-20T20:35:38.064 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json 2022-01-20T20:35:38.064 INFO:scrape:1 jobs: ['6624033'] 2022-01-20T20:35:38.064 INFO:scrape:suites: ['cluster/{12-osds', 'mon_election/classic', 'msgr-failures/fastclose', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{default}', 'supported-random-distro$/{rhel_8}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}'] 2022-01-20T20:35:38.065 INFO:scrape: