2023-04-11T09:13:24.111 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-11T09:13:24.112 INFO:teuthology.results:subset: '17/99' 2023-04-11T09:13:24.112 INFO:teuthology.results:seed: '2997' 2023-04-11T10:03:08.849 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-11T10:03:08.879 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-04-11T21:29:57.492 INFO:teuthology.results:Tests finished! gathering results... 2023-04-11T21:29:58.842 INFO:scrape:Found 16 jobs 2023-04-11T21:30:00.995 INFO:scrape:Found 4 distinct failure reasons 2023-04-11T21:30:00.996 INFO:scrape:Failure: Command failed on smithi145 with status 1: 'qemu-img convert -f qcow2 -O raw /home/ubuntu/cephtest/qemu/base.client.0.0.qcow2 rbd:rbd/client.0.0' 2023-04-11T21:30:00.997 INFO:scrape:12 jobs: ['7238149', '7238150', '7238148', '7238142', '7238147', '7238144', '7238156', '7238151', '7238146', '7238143', '7238153', '7238154'] 2023-04-11T21:30:00.997 INFO:scrape:suites intersection: ['openstack}'] 2023-04-11T21:30:00.997 INFO:scrape:suites union: ['2-cluster/{fix-2', '3-supported-random-distro$/{ubuntu_latest}', '4-cache-path', '5-cache-mode/rwl', '5-cache-mode/ssd', '6-cache-size/1G', '6-cache-size/5G', '7-workloads/qemu_xfstests}', 'clusters/{fixed-3', 'features/defaults', 'features/readbalance', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'pool/ec-data-pool', 'pool/none', 'pool/replicated-data-pool', 'pool/small-cache-pool', 'qemu/xfstests', 'rbd/maintenance/{base/install', 'rbd/pwl-cache/tmpfs/{1-base/install', 'rbd/qemu/{cache/none', 'rbd/qemu/{cache/writearound', 'rbd/qemu/{cache/writeback', 'rbd/qemu/{cache/writethrough', 'supported-random-distro$/{ubuntu_latest}', 'workloads/dynamic_features}', 'workloads/qemu_bonnie}', 'workloads/qemu_xfstests}', 'workloads/rebuild_object_map}'] 2023-04-11T21:30:00.998 INFO:scrape: 2023-04-11T21:30:01.000 INFO:scrape:Dead: b'2023-04-11T21:26:14.898 DEBUG:teuthology.exit:Finished running handlers' 2023-04-11T21:30:01.000 INFO:scrape:ceph version 18.0.0-2831-g2ddb2285 (2ddb228504b28af31004c13cb21971447794979e) reef (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f068d8ae420] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x55c84efc2558] 5: (KernelDevice::_aio_thread()+0xe41) [0x55c84fb538d1] 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55c84fb593a1] 7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f068d8a2609] 8: clone() 2023-04-11T21:30:01.000 INFO:scrape:1 jobs: ['7238145'] 2023-04-11T21:30:01.001 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{ubuntu_20.04}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}'] 2023-04-11T21:30:01.001 INFO:scrape: 2023-04-11T21:30:01.001 INFO:scrape:Failure: Command failed (workunit test rbd/qemu-iotests.sh) on smithi012 with status 13: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d11c6716ac738978d60e117685d7c1a8981052b8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/qemu-iotests.sh' 2023-04-11T21:30:01.002 INFO:scrape:1 jobs: ['7238152'] 2023-04-11T21:30:01.002 INFO:scrape:suites: ['objectstore/bluestore-hybrid', 'openstack', 'rbd/singleton/{all/qemu-iotests-writearound', 'supported-random-distro$/{ubuntu_latest}}'] 2023-04-11T21:30:01.002 INFO:scrape: 2023-04-11T21:30:01.003 INFO:scrape:Failure: reached maximum tries (30) after waiting for 300 seconds 2023-04-11T21:30:01.003 INFO:scrape:1 jobs: ['7238155'] 2023-04-11T21:30:01.003 INFO:scrape:suites: ['clusters/{fixed-3', 'features/defaults', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'openstack}', 'pool/ec-cache-pool', 'rbd/encryption/{cache/writeback', 'supported-random-distro$/{centos_8}', 'workloads/qemu_xfstests_none_luks1}'] 2023-04-11T21:30:01.003 INFO:scrape: 2023-04-11T21:30:01.004 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/ failed: 14 dead: 1 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [7238142] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/readbalance msgr-failures/few objectstore/bluestore-bitmap pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/qemu_xfstests} ----------------------------------------------------------------- time: 00:16:40 info: http://pulpito.ceph.com/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238142/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238142/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ebb0ee88460c4c3aa1b3cc1e0eab8873 Command failed on smithi145 with status 1: 'qemu-img convert -f qcow2 -O raw /home/ubuntu/cephtest/qemu/base.client.0.0.qcow2 rbd:rbd/client.0.0' [7238143] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-stupid qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/dynamic_features} ----------------------------------------------------------------- time: 00:15:46 info: http://pulpito.ceph.com/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238143/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238143/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7575ddfc9f0f4cfeaaecce9e55c594a8 Command failed on smithi115 with status 1: 'qemu-img convert -f qcow2 -O raw /home/ubuntu/cephtest/qemu/base.client.0.0.qcow2 rbd:rbd/client.0.0' [7238144] rbd/pwl-cache/tmpfs/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{ubuntu_latest} 4-cache-path 5-cache-mode/rwl 6-cache-size/1G 7-workloads/qemu_xfstests} ----------------------------------------------------------------- time: 00:14:20 info: http://pulpito.ceph.com/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238144/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238144/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e9e0fb37e6b2402a984d7e73682c3809 Command failed on smithi129 with status 1: 'qemu-img convert -f qcow2 -O raw /home/ubuntu/cephtest/qemu/base.client.0.0.qcow2 rbd:rbd/client.0.0' [7238146] rbd/qemu/{cache/writearound clusters/{fixed-3 openstack} features/readbalance msgr-failures/few objectstore/bluestore-comp-zlib pool/ec-data-pool supported-random-distro$/{ubuntu_latest} workloads/qemu_xfstests} ----------------------------------------------------------------- time: 00:12:51 info: http://pulpito.ceph.com/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238146/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238146/ sentry: https://sentry.ceph.com/organizations/ceph/?query=49fb5240015a496e89cba36a61d1dd07 Command failed on smithi141 with status 1: 'qemu-img convert -f qcow2 -O raw /home/ubuntu/cephtest/qemu/base.client.0.0.qcow2 rbd:rbd/client.0.0' [7238147] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-comp-snappy qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/dynamic_features} ----------------------------------------------------------------- time: 00:14:40 info: http://pulpito.ceph.com/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238147/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238147/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3427f04374dc4000a2e652d774689526 Command failed on smithi143 with status 1: 'qemu-img convert -f qcow2 -O raw /home/ubuntu/cephtest/qemu/base.client.0.0.qcow2 rbd:rbd/client.0.0' [7238148] rbd/pwl-cache/tmpfs/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{ubuntu_latest} 4-cache-path 5-cache-mode/ssd 6-cache-size/5G 7-workloads/qemu_xfstests} ----------------------------------------------------------------- time: 00:13:59 info: http://pulpito.ceph.com/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238148/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238148/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aefb885039284d5f9a3d7701c33612fc Command failed on smithi115 with status 1: 'qemu-img convert -f qcow2 -O raw /home/ubuntu/cephtest/qemu/base.client.0.0.qcow2 rbd:rbd/client.0.0' [7238149] rbd/qemu/{cache/none clusters/{fixed-3 openstack} features/readbalance msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/qemu_xfstests} ----------------------------------------------------------------- time: 00:14:46 info: http://pulpito.ceph.com/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238149/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238149/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e1bb3aa6f46649b9825f897fd6bff628 Command failed on smithi145 with status 1: 'qemu-img convert -f qcow2 -O raw /home/ubuntu/cephtest/qemu/base.client.0.0.qcow2 rbd:rbd/client.0.0' [7238150] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-comp-zstd qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/rebuild_object_map} ----------------------------------------------------------------- time: 00:14:10 info: http://pulpito.ceph.com/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238150/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238150/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bec1c996bec8410f9836e2eec98653df Command failed on smithi191 with status 1: 'qemu-img convert -f qcow2 -O raw /home/ubuntu/cephtest/qemu/base.client.0.0.qcow2 rbd:rbd/client.0.0' [7238151] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-hybrid qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/dynamic_features} ----------------------------------------------------------------- time: 00:12:53 info: http://pulpito.ceph.com/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238151/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238151/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f510aab7e949468ab1dba524e2543b93 Command failed on smithi129 with status 1: 'qemu-img convert -f qcow2 -O raw /home/ubuntu/cephtest/qemu/base.client.0.0.qcow2 rbd:rbd/client.0.0' [7238152] rbd/singleton/{all/qemu-iotests-writearound objectstore/bluestore-hybrid openstack supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:12:02 info: http://pulpito.ceph.com/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238152/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238152/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8529233efffe487ab7d8e77539099469 Command failed (workunit test rbd/qemu-iotests.sh) on smithi012 with status 13: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d11c6716ac738978d60e117685d7c1a8981052b8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/qemu-iotests.sh' [7238153] rbd/qemu/{cache/writethrough clusters/{fixed-3 openstack} features/readbalance msgr-failures/few objectstore/bluestore-comp-lz4 pool/none supported-random-distro$/{ubuntu_latest} workloads/qemu_xfstests} ----------------------------------------------------------------- time: 00:12:32 info: http://pulpito.ceph.com/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238153/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238153/ sentry: https://sentry.ceph.com/organizations/ceph/?query=895a2b4b15b2419398dca5edb0778c5a Command failed on smithi141 with status 1: 'qemu-img convert -f qcow2 -O raw /home/ubuntu/cephtest/qemu/base.client.0.0.qcow2 rbd:rbd/client.0.0' [7238154] rbd/qemu/{cache/none clusters/{fixed-3 openstack} features/defaults msgr-failures/few objectstore/bluestore-comp-snappy pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/qemu_bonnie} ----------------------------------------------------------------- time: 00:12:50 info: http://pulpito.ceph.com/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238154/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238154/ sentry: https://sentry.ceph.com/organizations/ceph/?query=285024415b6c495dbfc5b6017e3853ee Command failed on smithi156 with status 1: 'qemu-img convert -f qcow2 -O raw /home/ubuntu/cephtest/qemu/base.client.0.0.qcow2 rbd:rbd/client.0.0' [7238155] rbd/encryption/{cache/writeback clusters/{fixed-3 openstack} features/defaults msgr-failures/few objectstore/bluestore-comp-snappy pool/ec-cache-pool supported-random-distro$/{centos_8} workloads/qemu_xfstests_none_luks1} ----------------------------------------------------------------- time: 02:53:14 info: http://pulpito.ceph.com/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238155/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238155/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3cffd6a093164f7389a704f9e3bca099 reached maximum tries (30) after waiting for 300 seconds [7238156] rbd/pwl-cache/tmpfs/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{ubuntu_latest} 4-cache-path 5-cache-mode/rwl 6-cache-size/5G 7-workloads/qemu_xfstests} ----------------------------------------------------------------- time: 00:12:15 info: http://pulpito.ceph.com/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238156/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238156/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e2599b9843ff4ffdaaff71c4d976867d Command failed on smithi176 with status 1: 'qemu-img convert -f qcow2 -O raw /home/ubuntu/cephtest/qemu/base.client.0.0.qcow2 rbd:rbd/client.0.0' Dead ================================================================= [7238145] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_20.04} workloads/rbd-mirror-snapshot-stress-workunit-minimum} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238145/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238145/ hit max job timeout Pass ================================================================= [7238141] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} time: 01:52:40 info: http://pulpito.ceph.com/dis-2023-04-11_09:07:09-rbd-wip-rbd-reef-backports-1-distro-default-smithi/7238141/