2024-03-16T10:03:27.058 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-16T10:03:27.058 INFO:teuthology.results:subset: '96/99' 2024-03-16T10:03:27.058 INFO:teuthology.results:seed: '4958' 2024-03-16T10:15:44.829 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-16T10:15:44.832 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-16T22:16:00.114 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2024-03-16T22:16:00.850 INFO:scrape:Found 10 jobs 2024-03-16T22:16:01.295 INFO:scrape:Found 2 distinct failure reasons 2024-03-16T22:16:01.296 INFO:scrape:Crash: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi186 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d146d0be924e60f5196e8f1cd52899abf176dfb7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2024-03-16T22:16:01.296 INFO:scrape:ceph version 19.0.0-1674-gd146d0be (d146d0be924e60f5196e8f1cd52899abf176dfb7) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff53ca3d520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x559323eea7a7] 6: (KernelDevice::_aio_thread()+0x9e2) [0x559324889db2] 7: ceph-osd(+0xd97681) [0x55932488a681] 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff53ca8fb43] 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff53cb21a00] 2024-03-16T22:16:01.296 INFO:scrape:1 jobs: ['7606188'] 2024-03-16T22:16:01.296 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}'] 2024-03-16T22:16:01.296 INFO:scrape: 2024-03-16T22:16:01.296 INFO:scrape:Dead: b'2024-03-16T22:13:17.267 DEBUG:teuthology.exit:Finished running handlers' 2024-03-16T22:16:01.296 INFO:scrape:ceph version 19.0.0-1674-gd146d0be (d146d0be924e60f5196e8f1cd52899abf176dfb7) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f118c654db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f118c6a154c] 3: raise() 4: abort() 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x558fc28c96af] 6: (KernelDevice::_aio_thread()+0x11e0) [0x558fc32915d0] 7: ceph-osd(+0xdbb741) [0x558fc3291741] 8: /lib64/libc.so.6(+0x9f802) [0x7f118c69f802] 9: /lib64/libc.so.6(+0x3f450) [0x7f118c63f450] 2024-03-16T22:16:01.297 INFO:scrape:1 jobs: ['7606186'] 2024-03-16T22:16:01.297 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_latest}', 'workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}'] 2024-03-16T22:16:01.297 INFO:scrape: 2024-03-16T22:16:01.298 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2024-03-16_09:58:05-rbd-wip-dis-testing-squid-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/dis-2024-03-16_09:58:05-rbd-wip-dis-testing-squid-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2024-03-16_09:58:05-rbd-wip-dis-testing-squid-distro-default-smithi/ failed: 1 dead: 0 running: 1 waiting: 0 queued: 0 passed: 8 Fail ================================================================= [7606188] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} ----------------------------------------------------------------- time: 04:51:24 info: https://pulpito.ceph.com/dis-2024-03-16_09:58:05-rbd-wip-dis-testing-squid-distro-default-smithi/7606188/ log: http://qa-proxy.ceph.com/teuthology/dis-2024-03-16_09:58:05-rbd-wip-dis-testing-squid-distro-default-smithi/7606188/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e26ff3e0ffe44eacadb73608c309e1a0 Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi186 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d146d0be924e60f5196e8f1cd52899abf176dfb7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' Running ================================================================= [7606186] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} info: https://pulpito.ceph.com/dis-2024-03-16_09:58:05-rbd-wip-dis-testing-squid-distro-default-smithi/7606186/ Pass ================================================================= [7606181] rbd/iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} time: 00:22:41 info: https://pulpito.ceph.com/dis-2024-03-16_09:58:05-rbd-wip-dis-testing-squid-distro-default-smithi/7606181/ [7606182] rbd/pwl-cache/home/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{ubuntu_latest} 4-cache-path 5-cache-mode/rwl 6-cache-size/1G 7-workloads/c_api_tests_with_defaults conf/{disable-pool-app}} time: 01:19:00 info: https://pulpito.ceph.com/dis-2024-03-16_09:58:05-rbd-wip-dis-testing-squid-distro-default-smithi/7606182/ [7606183] rbd/maintenance/{base/install clusters/{fixed-3 openstack} conf/{disable-pool-app} objectstore/bluestore-comp-zstd qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/dynamic_features} time: 02:09:38 info: https://pulpito.ceph.com/dis-2024-03-16_09:58:05-rbd-wip-dis-testing-squid-distro-default-smithi/7606183/ [7606184] rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/none features/journaling msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_latest} workloads/rbd_cli_generic} time: 00:26:27 info: https://pulpito.ceph.com/dis-2024-03-16_09:58:05-rbd-wip-dis-testing-squid-distro-default-smithi/7606184/ [7606185] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/replicated extra-conf/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_latest} workloads/c_api_tests} time: 00:40:40 info: https://pulpito.ceph.com/dis-2024-03-16_09:58:05-rbd-wip-dis-testing-squid-distro-default-smithi/7606185/ [7606187] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_latest} workloads/rbd-mirror-workunit-policy-none} time: 00:31:01 info: https://pulpito.ceph.com/dis-2024-03-16_09:58:05-rbd-wip-dis-testing-squid-distro-default-smithi/7606187/ [7606189] rbd/immutable-object-cache/{clusters/{fix-2 openstack} conf/{disable-pool-app} pool/ceph_and_immutable_object_cache supported-random-distro$/{ubuntu_latest} workloads/qemu_on_immutable_object_cache_and_thrash} time: 01:46:10 info: https://pulpito.ceph.com/dis-2024-03-16_09:58:05-rbd-wip-dis-testing-squid-distro-default-smithi/7606189/ [7606190] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/none features/journaling msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/qemu_bonnie} time: 00:24:47 info: https://pulpito.ceph.com/dis-2024-03-16_09:58:05-rbd-wip-dis-testing-squid-distro-default-smithi/7606190/