2023-07-07T12:18:32.448 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-07T12:18:32.449 INFO:teuthology.results:subset: '111/120000' 2023-07-07T12:18:32.449 INFO:teuthology.results:seed: '351' 2023-07-07T12:19:16.373 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-07T12:19:16.377 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-07-08T00:20:02.877 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-07-08T00:20:03.278 INFO:scrape:Found 5 jobs 2023-07-08T00:20:04.516 INFO:scrape:Found 4 distinct failure reasons 2023-07-08T00:20:04.517 INFO:scrape:Dead: b'2023-07-08T00:20:02.534 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-08T00:20:02.533+0000 7fe977d2b700 -1 osd.1 25 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.4230.0:6640 3.2 3:53a0b873:::rbd_data.1083d907f8d6.0000000000000015:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1404928~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e24)' 2023-07-08T00:20:04.517 INFO:scrape:1 jobs: ['7328361'] 2023-07-08T00:20:04.517 INFO:scrape:suites: ['clusters/{fixed-1', 'features/layering', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'openstack}', 'pool/small-cache-pool', 'rbd/cli/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd_cli_migration}'] 2023-07-08T00:20:04.518 INFO:scrape: 2023-07-08T00:20:04.518 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/diff_continuous.sh 2023-07-08T00:20:04.518 INFO:scrape:1 jobs: ['7328362'] 2023-07-08T00:20:04.518 INFO:scrape:suites: ['cluster/{fixed-3', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'openstack}', 'rbd/nbd/{base/install', 'supported-random-distro$/{centos_8}', 'thrashers/cache', 'thrashosds-health', 'workloads/rbd_nbd_diff_continuous}'] 2023-07-08T00:20:04.519 INFO:scrape: 2023-07-08T00:20:04.519 INFO:scrape:Failure: Command failed on smithi007 with status 1: 'CEPH_REF=master CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /home/ubuntu/cephtest/virtualenv/bin/cram -v -- /home/ubuntu/cephtest/archive/cram.client.0/*.t' 2023-07-08T00:20:04.519 INFO:scrape:1 jobs: ['7328363'] 2023-07-08T00:20:04.520 INFO:scrape:suites: ['base/install', 'cluster/{fixed-3', 'openstack}', 'rbd/iscsi/{0-single-container-host', 'workloads/cephadm_iscsi}'] 2023-07-08T00:20:04.520 INFO:scrape: 2023-07-08T00:20:04.520 INFO:scrape:Dead: b"2023-07-07T23:41:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ rbd --cluster cluster1 -p mirror --namespace '' ls" 2023-07-08T00:20:04.520 INFO:scrape:ceph version 18.1.0-241-gb75c5422 (b75c542210c04254e2c94e13c158a5b74292e5f0) reef (rc) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fc20d1a9cf0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55d727d8fd81] 5: (KernelDevice::_aio_thread()+0x1798) [0x55d72899e888] 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55d7289a0f21] 7: /lib64/libpthread.so.0(+0x81ca) [0x7fc20d19f1ca] 8: clone() 2023-07-08T00:20:04.521 INFO:scrape:1 jobs: ['7328365'] 2023-07-08T00:20:04.521 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}'] 2023-07-08T00:20:04.521 INFO:scrape: 2023-07-08T00:20:04.522 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2023-07-07_12:17:50-rbd-wip-yuri11-testing-2023-06-27-0812-reef-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/dis-2023-07-07_12:17:50-rbd-wip-yuri11-testing-2023-06-27-0812-reef-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2023-07-07_12:17:50-rbd-wip-yuri11-testing-2023-06-27-0812-reef-distro-default-smithi/ failed: 2 dead: 0 running: 2 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [7328362] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8} thrashers/cache thrashosds-health workloads/rbd_nbd_diff_continuous} ----------------------------------------------------------------- time: 03:23:33 info: http://pulpito.ceph.com/dis-2023-07-07_12:17:50-rbd-wip-yuri11-testing-2023-06-27-0812-reef-distro-default-smithi/7328362/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-07_12:17:50-rbd-wip-yuri11-testing-2023-06-27-0812-reef-distro-default-smithi/7328362/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a9e0714a5a4c48209af21a2d1ba712c6 Command failed (workunit test rbd/diff_continuous.sh) on smithi083 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b75c542210c04254e2c94e13c158a5b74292e5f0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' [7328363] rbd/iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_iscsi} ----------------------------------------------------------------- time: 00:32:20 info: http://pulpito.ceph.com/dis-2023-07-07_12:17:50-rbd-wip-yuri11-testing-2023-06-27-0812-reef-distro-default-smithi/7328363/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-07-07_12:17:50-rbd-wip-yuri11-testing-2023-06-27-0812-reef-distro-default-smithi/7328363/ sentry: https://sentry.ceph.com/organizations/ceph/?query=00e4d7f2567d400d97137f862f6ea719 Command failed on smithi007 with status 1: 'CEPH_REF=master CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /home/ubuntu/cephtest/virtualenv/bin/cram -v -- /home/ubuntu/cephtest/archive/cram.client.0/*.t' Running ================================================================= [7328361] rbd/cli/{base/install clusters/{fixed-1 openstack} features/layering msgr-failures/few objectstore/bluestore-comp-zstd pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_cli_migration} info: http://pulpito.ceph.com/dis-2023-07-07_12:17:50-rbd-wip-yuri11-testing-2023-06-27-0812-reef-distro-default-smithi/7328361/ [7328365] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} info: http://pulpito.ceph.com/dis-2023-07-07_12:17:50-rbd-wip-yuri11-testing-2023-06-27-0812-reef-distro-default-smithi/7328365/ Pass ================================================================= [7328364] rbd/pwl-cache/tmpfs/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{rhel_8} 4-cache-path 5-cache-mode/ssd 6-cache-size/5G 7-workloads/qemu_xfstests} time: 01:27:16 info: http://pulpito.ceph.com/dis-2023-07-07_12:17:50-rbd-wip-yuri11-testing-2023-06-27-0812-reef-distro-default-smithi/7328364/