2023-07-21T17:01:01.871 INFO:root:teuthology version: 0.0.1.dev153+g79491cb
2023-07-21T17:01:01.873 INFO:teuthology.results:subset: '111/120000'
2023-07-21T17:01:01.873 INFO:teuthology.results:seed: '4600'
2023-07-21T21:19:21.485 INFO:root:teuthology version: 0.0.1.dev153+g79491cb
2023-07-21T21:19:21.511 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-07-22T09:19:34.513 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2023-07-22T09:19:37.880 INFO:scrape:Found 40 jobs
2023-07-22T09:19:38.882 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2023-07-21_15:23:26-rbd-wip-yuri7-testing-2023-07-20-0727-distro-default-smithi/7346807/teuthology.log
2023-07-22T09:19:38.883 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2023-07-21_15:23:26-rbd-wip-yuri7-testing-2023-07-20-0727-distro-default-smithi/7346807/teuthology.log
2023-07-22T09:19:40.611 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2023-07-21_15:23:26-rbd-wip-yuri7-testing-2023-07-20-0727-distro-default-smithi/7346742/teuthology.log
2023-07-22T09:19:41.300 INFO:scrape:Found 8 distinct failure reasons
2023-07-22T09:19:41.301 INFO:scrape:Failure: Command failed (workunit test rbd/test_librbd.sh) on smithi157 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=abb63cf06fecdaf9d8a79c11761a4d81cad3fd0a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=125 VALGRIND=\'--tool=memcheck --leak-check=full\' adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh'
2023-07-22T09:19:41.301 INFO:scrape:3 jobs: ['7346786', '7346756', '7346767']
2023-07-22T09:19:41.302 INFO:scrape:suites intersection: ['centos_latest', 'clusters/{fixed-1', 'openstack}', 'rbd/valgrind/{base/install', 'validator/memcheck']
2023-07-22T09:19:41.302 INFO:scrape:suites union: ['centos_latest', 'clusters/{fixed-1', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rbd/valgrind/{base/install', 'validator/memcheck', 'workloads/c_api_tests_with_defaults}', 'workloads/c_api_tests_with_journaling}', 'workloads/c_api_tests}']
2023-07-22T09:19:41.302 INFO:scrape:
2023-07-22T09:19:41.302 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/diff_continuous.sh
2023-07-22T09:19:41.303 INFO:scrape:2 jobs: ['7346724', '7346795']
2023-07-22T09:19:41.303 INFO:scrape:suites intersection: ['cluster/{fixed-3', 'msgr-failures/few', 'openstack}', 'rbd/nbd/{base/install', 'thrashers/cache', 'thrashosds-health', 'workloads/rbd_nbd_diff_continuous}']
2023-07-22T09:19:41.303 INFO:scrape:suites union: ['cluster/{fixed-3', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rbd/nbd/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/cache', 'thrashosds-health', 'workloads/rbd_nbd_diff_continuous}']
2023-07-22T09:19:41.304 INFO:scrape:
2023-07-22T09:19:41.304 INFO:scrape:Failure: Command failed (workunit test rbd/test_librbd_python.sh) on smithi045 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=abb63cf06fecdaf9d8a79c11761a4d81cad3fd0a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 VALGRIND=\'--tool=memcheck --leak-check=full\' adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh'
2023-07-22T09:19:41.304 INFO:scrape:4 jobs: ['7346715', '7346707', '7346713', '7346733']
2023-07-22T09:19:41.305 INFO:scrape:suites intersection: ['clusters/{fixed-1', 'openstack}']
2023-07-22T09:19:41.305 INFO:scrape:suites union: ['cachepool/none', 'centos_latest', 'clusters/{fixed-1', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rbd/basic/{base/install', 'rbd/valgrind/{base/install', 'supported-random-distro$/{centos_latest}', 'tasks/rbd_python_api_tests_old_format}', 'validator/memcheck', 'workloads/python_api_tests_with_defaults}', 'workloads/python_api_tests_with_journaling}', 'workloads/python_api_tests}']
2023-07-22T09:19:41.305 INFO:scrape:
2023-07-22T09:19:41.305 INFO:scrape:Dead: b'2023-07-22T09:14:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi082.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls -l'
2023-07-22T09:19:41.306 INFO:scrape:ceph version 18.0.0-5036-gabb63cf0 (abb63cf06fecdaf9d8a79c11761a4d81cad3fd0a) reef (dev)
 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fa166323ce0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x197) [0x5555a55d172f]
 5: (KernelDevice::_aio_thread()+0x1686) [0x5555a61f07f6]
 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5555a61f62b1]
 7: /lib64/libpthread.so.0(+0x81cf) [0x7fa1663191cf]
 8: clone()
2023-07-22T09:19:41.306 INFO:scrape:1 jobs: ['7346810']
2023-07-22T09:19:41.306 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{rhel_8}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}']
2023-07-22T09:19:41.307 INFO:scrape:
2023-07-22T09:19:41.307 INFO:scrape:Dead: None
2023-07-22T09:19:41.307 INFO:scrape:2 jobs: ['7346807', '7346742']
2023-07-22T09:19:41.307 INFO:scrape:
2023-07-22T09:19:41.308 INFO:scrape:Dead: b'2023-07-22T05:15:32.679 DEBUG:teuthology.exit:Finished running handlers'
2023-07-22T09:19:41.308 INFO:scrape:1 jobs: ['7346718']
2023-07-22T09:19:41.308 INFO:scrape:suites: ['clusters/{fixed-1', 'features/layering', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'openstack}', 'pool/small-cache-pool', 'rbd/cli/{base/install', 'supported-random-distro$/{rhel_8}', 'workloads/rbd_cli_migration}']
2023-07-22T09:19:41.309 INFO:scrape:
2023-07-22T09:19:41.309 INFO:scrape:Failure: Command failed (workunit test rbd/test_rbd_mirror.sh) on smithi012 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=abb63cf06fecdaf9d8a79c11761a4d81cad3fd0a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 VALGRIND=\'--tool=memcheck --leak-check=full\' adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_rbd_mirror.sh'
2023-07-22T09:19:41.309 INFO:scrape:1 jobs: ['7346746']
2023-07-22T09:19:41.309 INFO:scrape:suites: ['centos_latest', 'clusters/{fixed-1', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rbd/valgrind/{base/install', 'validator/memcheck', 'workloads/rbd_mirror}']
2023-07-22T09:19:41.310 INFO:scrape:
2023-07-22T09:19:41.310 INFO:scrape:Failure: Command failed on smithi097 with status 1: 'mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log'
2023-07-22T09:19:41.310 INFO:scrape:1 jobs: ['7346770']
2023-07-22T09:19:41.311 INFO:scrape:suites: ['clusters/{fixed-2', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'openstack}', 'rbd/thrash/{base/install', 'supported-random-distro$/{centos_latest}', 'thrashers/cache', 'thrashosds-health', 'workloads/rbd_api_tests_copy_on_read}']
2023-07-22T09:19:41.311 INFO:scrape: