2023-10-17T06:43:40.986 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-17T06:43:40.987 INFO:teuthology.results:subset: '49/99' 2023-10-17T06:43:40.987 INFO:teuthology.results:seed: '2449' 2023-10-17T07:51:50.496 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-17T07:51:50.499 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-17T12:49:12.725 INFO:teuthology.results:Tests finished! gathering results... 2023-10-17T12:49:13.494 INFO:scrape:Found 10 jobs 2023-10-17T12:49:13.570 INFO:scrape:Found 1 distinct failure reasons 2023-10-17T12:49:13.570 INFO:scrape:Crash: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi130 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=735f68d7bf91d643813a8420fc2b11441f4bd8c4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2023-10-17T12:49:13.571 INFO:scrape:ceph version 18.0.0-6724-g01fff6a7 (01fff6a72a328459c1d153e5dc1de6a34e48a82f) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff61ae73520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x5638c90f880d] 6: (KernelDevice::_aio_thread()+0x9da) [0x5638c99d01ea] 7: ceph-osd(+0x15acab1) [0x5638c99d0ab1] 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff61aec5b43] 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff61af57a00] 2023-10-17T12:49:13.571 INFO:scrape:1 jobs: ['7431059'] 2023-10-17T12:49:13.572 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}'] 2023-10-17T12:49:13.572 INFO:scrape: 2023-10-17T12:49:13.573 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2023-10-17_06:43:15-rbd-wip-dis-testing-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/dis-2023-10-17_06:43:15-rbd-wip-dis-testing-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2023-10-17_06:43:15-rbd-wip-dis-testing-distro-default-smithi/ failed: 1 dead: 0 running: 0 waiting: 0 queued: 0 passed: 9 Fail ================================================================= [7431059] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} ----------------------------------------------------------------- time: 04:51:39 info: http://pulpito.ceph.com/dis-2023-10-17_06:43:15-rbd-wip-dis-testing-distro-default-smithi/7431059/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-10-17_06:43:15-rbd-wip-dis-testing-distro-default-smithi/7431059/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8c0898194b3641968b43ab55683b78a6 Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi130 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=735f68d7bf91d643813a8420fc2b11441f4bd8c4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' Pass ================================================================= [7431053] rbd/nbd/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} time: 00:36:22 info: http://pulpito.ceph.com/dis-2023-10-17_06:43:15-rbd-wip-dis-testing-distro-default-smithi/7431053/ [7431054] rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/none features/journaling msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} time: 01:50:14 info: http://pulpito.ceph.com/dis-2023-10-17_06:43:15-rbd-wip-dis-testing-distro-default-smithi/7431054/ [7431055] rbd/immutable-object-cache/{clusters/{fix-2 openstack} conf/{disable-pool-app} pool/ceph_and_immutable_object_cache supported-random-distro$/{centos_latest} workloads/c_api_tests_with_defaults} time: 00:43:27 info: http://pulpito.ceph.com/dis-2023-10-17_06:43:15-rbd-wip-dis-testing-distro-default-smithi/7431055/ [7431056] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/replicated extra-conf/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_defaults} time: 00:50:52 info: http://pulpito.ceph.com/dis-2023-10-17_06:43:15-rbd-wip-dis-testing-distro-default-smithi/7431056/ [7431057] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/ec extra-conf/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_latest} workloads/c_api_tests} time: 00:42:43 info: http://pulpito.ceph.com/dis-2023-10-17_06:43:15-rbd-wip-dis-testing-distro-default-smithi/7431057/ [7431058] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/none extra-conf/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_latest} workloads/c_api_tests_with_defaults} time: 00:44:26 info: http://pulpito.ceph.com/dis-2023-10-17_06:43:15-rbd-wip-dis-testing-distro-default-smithi/7431058/ [7431060] rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/none features/journaling msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} time: 01:53:44 info: http://pulpito.ceph.com/dis-2023-10-17_06:43:15-rbd-wip-dis-testing-distro-default-smithi/7431060/ [7431061] rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} time: 01:50:48 info: http://pulpito.ceph.com/dis-2023-10-17_06:43:15-rbd-wip-dis-testing-distro-default-smithi/7431061/ [7431062] rbd/pwl-cache/tmpfs/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{ubuntu_latest} 4-cache-path 5-cache-mode/ssd 6-cache-size/1G 7-workloads/qemu_xfstests conf/{disable-pool-app}} time: 01:17:18 info: http://pulpito.ceph.com/dis-2023-10-17_06:43:15-rbd-wip-dis-testing-distro-default-smithi/7431062/