2023-08-17T23:37:11.060 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-17T23:37:11.061 INFO:teuthology.results:subset: '111/120000' 2023-08-17T23:37:11.061 INFO:teuthology.results:seed: '2311' 2023-08-17T23:40:27.418 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-17T23:40:27.422 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-08-18T11:41:13.495 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-08-18T11:41:14.387 INFO:scrape:Found 11 jobs 2023-08-18T11:41:16.038 INFO:scrape:Found 5 distinct failure reasons 2023-08-18T11:41:16.039 INFO:scrape:Failure: "2023-08-18T00:00:04.610614+0000 mon.a (mon.0) 179 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log 2023-08-18T11:41:16.039 INFO:scrape:1 jobs: ['7372355'] 2023-08-18T11:41:16.040 INFO:scrape:suites: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_latest}', 'workloads/rbd-mirror-workunit-policy-none}'] 2023-08-18T11:41:16.040 INFO:scrape: 2023-08-18T11:41:16.040 INFO:scrape:Dead: b'2023-08-18T11:41:14.128 INFO:tasks.ceph.osd.0.smithi100.stderr:2023-08-18T11:41:14.121+0000 7f42cd075640 -1 osd.0 26 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.4223.0:8531 3.3 3:c50d2619:::rbd_data.107cbb11b98b.00000000000000d1:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2351104~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)' 2023-08-18T11:41:16.041 INFO:scrape:1 jobs: ['7372353'] 2023-08-18T11:41:16.041 INFO:scrape:suites: ['clusters/{fixed-1', 'features/layering', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'openstack}', 'pool/small-cache-pool', 'rbd/cli/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd_cli_migration}'] 2023-08-18T11:41:16.041 INFO:scrape: 2023-08-18T11:41:16.041 INFO:scrape:Dead: b"2023-08-18T11:03:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi157.stderr:+++ rbd --cluster cluster1 -p mirror --namespace '' ls" 2023-08-18T11:41:16.042 INFO:scrape:ceph version 18.2.0-103-g98da3bc8 (98da3bc80b7f94a387fd066563cc3beb2e965ebe) reef (stable) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7faf03c92ce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55bcb767ad81] 5: (KernelDevice::_aio_thread()+0x1798) [0x55bcb828a078] 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55bcb828c711] 7: /lib64/libpthread.so.0(+0x81cf) [0x7faf03c881cf] 8: clone() 2023-08-18T11:41:16.042 INFO:scrape:1 jobs: ['7372362'] 2023-08-18T11:41:16.042 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{rhel_8}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}'] 2023-08-18T11:41:16.043 INFO:scrape: 2023-08-18T11:41:16.043 INFO:scrape:Failure: Command failed on smithi116 with status 1: 'test -f /home/ubuntu/cephtest/archive/qemu/client.0/success' 2023-08-18T11:41:16.043 INFO:scrape:1 jobs: ['7372357'] 2023-08-18T11:41:16.043 INFO:scrape:suites: ['2-cluster/{fix-2', '3-supported-random-distro$/{ubuntu_latest}', '4-cache-path', '5-cache-mode/ssd', '6-cache-size/5G', '7-workloads/qemu_xfstests}', 'openstack}', 'rbd/pwl-cache/tmpfs/{1-base/install'] 2023-08-18T11:41:16.044 INFO:scrape: 2023-08-18T11:41:16.044 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/diff_continuous.sh 2023-08-18T11:41:16.044 INFO:scrape:2 jobs: ['7372359', '7372354'] 2023-08-18T11:41:16.045 INFO:scrape:suites intersection: ['cluster/{fixed-3', 'msgr-failures/few', 'openstack}', 'rbd/nbd/{base/install', 'thrashers/cache', 'thrashosds-health', 'workloads/rbd_nbd_diff_continuous}'] 2023-08-18T11:41:16.045 INFO:scrape:suites union: ['cluster/{fixed-3', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rbd/nbd/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_20.04}', 'thrashers/cache', 'thrashosds-health', 'workloads/rbd_nbd_diff_continuous}'] 2023-08-18T11:41:16.045 INFO:scrape: