2023-11-27T11:31:50.914 INFO:root:teuthology version: 0.0.1.dev205+g2442239 2023-11-27T11:31:50.914 INFO:teuthology.results:subset: '14/99' 2023-11-27T11:31:50.914 INFO:teuthology.results:seed: '2662' 2023-11-27T11:32:39.550 INFO:root:teuthology version: 0.0.1.dev205+g2442239 2023-11-27T11:32:39.554 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-27T16:32:01.536 INFO:teuthology.results:Tests finished! gathering results... 2023-11-27T16:32:02.446 INFO:scrape:Found 12 jobs 2023-11-27T16:32:03.189 INFO:scrape:Found 3 distinct failure reasons 2023-11-27T16:32:03.189 INFO:scrape:Failure: Command failed on smithi192 with status 1: 'test -f /home/ubuntu/cephtest/archive/qemu/client.0/success' 2023-11-27T16:32:03.189 INFO:scrape:1 jobs: ['7467568'] 2023-11-27T16:32:03.189 INFO:scrape:suites: ['2-cluster/{fix-2', '3-supported-random-distro$/{ubuntu_latest}', '4-cache-path', '5-cache-mode/rwl', '6-cache-size/1G', '7-workloads/qemu_xfstests', 'conf/{disable-pool-app}}', 'openstack}', 'rbd/pwl-cache/tmpfs/{1-base/install'] 2023-11-27T16:32:03.189 INFO:scrape: 2023-11-27T16:32:03.189 INFO:scrape:Crash: Command failed on smithi183 with status 139: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'echo \'"\'"\'{"type":"qcow","stream":{"type":"http","url":"http://download.ceph.com/qa/ubuntu-12.04.qcow2"}}\'"\'"\' | rbd migration prepare --import-only --source-spec-path - client.0.0\'' 2023-11-27T16:32:03.190 INFO:scrape:ceph version 18.0.0-7516-g33df3bc2 (33df3bc2b6d55e42c1e87ab8d6f18392df19d216) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f8a7d454db0] 2: /lib64/librbd.so.1(+0x2f5c31) [0x7f8a7eaf5c31] 3: /lib64/librados.so.2(+0x12f65e) [0x7f8a7e77865e] 4: /lib64/librados.so.2(+0xcc1cf) [0x7f8a7e7151cf] 5: /lib64/libstdc++.so.6(+0xdb924) [0x7f8a7d8db924] 6: /lib64/libc.so.6(+0x9f802) [0x7f8a7d49f802] 7: /lib64/libc.so.6(+0x3f450) [0x7f8a7d43f450] 2023-11-27T16:32:03.190 INFO:scrape:3 jobs: ['7467574', '7467572', '7467567'] 2023-11-27T16:32:03.190 INFO:scrape:suites intersection: ['2-clusters/{fixed-3', '4-supported-random-distro$/{centos_latest}', '5-data-pool/replicated', '6-prepare/qcow2-http', '7-io-workloads/qemu_xfstests', '8-migrate-workloads/execute', '9-cleanup/cleanup', 'conf/{disable-pool-app}}', 'openstack}', 'rbd/migration/{1-base/install'] 2023-11-27T16:32:03.190 INFO:scrape:suites union: ['2-clusters/{fixed-3', '3-objectstore/bluestore-bitmap', '3-objectstore/bluestore-comp-zlib', '3-objectstore/bluestore-hybrid', '4-supported-random-distro$/{centos_latest}', '5-data-pool/replicated', '6-prepare/qcow2-http', '7-io-workloads/qemu_xfstests', '8-migrate-workloads/execute', '9-cleanup/cleanup', 'conf/{disable-pool-app}}', 'openstack}', 'rbd/migration/{1-base/install'] 2023-11-27T16:32:03.190 INFO:scrape: 2023-11-27T16:32:03.190 INFO:scrape:Crash: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi176 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=33df3bc2b6d55e42c1e87ab8d6f18392df19d216 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2023-11-27T16:32:03.190 INFO:scrape:ceph version 18.0.0-7516-g33df3bc2 (33df3bc2b6d55e42c1e87ab8d6f18392df19d216) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f77f6854db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f77f68a154c] 3: raise() 4: abort() 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x5573d6feeefc] 6: (KernelDevice::_aio_thread()+0x10f6) [0x5573d793dda6] 7: ceph-osd(+0xd9ff91) [0x5573d791ef91] 8: /lib64/libc.so.6(+0x9f802) [0x7f77f689f802] 9: /lib64/libc.so.6(+0x3f450) [0x7f77f683f450] 2023-11-27T16:32:03.190 INFO:scrape:1 jobs: ['7467575'] 2023-11-27T16:32:03.190 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_latest}', 'workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}'] 2023-11-27T16:32:03.190 INFO:scrape: 2023-11-27T16:32:03.191 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [7467567] rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-hybrid 4-supported-random-distro$/{centos_latest} 5-data-pool/replicated 6-prepare/qcow2-http 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup conf/{disable-pool-app}} ----------------------------------------------------------------- time: 00:10:06 info: https://pulpito.ceph.com/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/7467567/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/7467567/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e304a1c236d4484b9f57193ce0ead534 Command failed on smithi137 with status 139: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'echo \'"\'"\'{"type":"qcow","stream ":{"type":"http","url":"http://download.ceph.com/qa/ubuntu-12.04.qcow2"}}\' "\'"\' | rbd migration prepare --import-only --source-spec-path - client.0.0\'' [7467568] rbd/pwl-cache/tmpfs/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{ubuntu_latest} 4-cache-path 5-cache-mode/rwl 6-cache-size/1G 7-workloads/qemu_xfstests conf/{disable-pool-app}} ----------------------------------------------------------------- time: 01:30:23 info: https://pulpito.ceph.com/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/7467568/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/7467568/ Command failed on smithi192 with status 1: 'test -f /home/ubuntu/cephtest/archive/qemu/client.0/success' [7467572] rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-bitmap 4-supported-random-distro$/{centos_latest} 5-data-pool/replicated 6-prepare/qcow2-http 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup conf/{disable-pool-app}} ----------------------------------------------------------------- time: 00:09:57 info: https://pulpito.ceph.com/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/7467572/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/7467572/ sentry: https://sentry.ceph.com/organizations/ceph/?query=69486272b3a14588b793c5688c8e6dbf Command failed on smithi204 with status 139: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'echo \'"\'"\'{"type":"qcow","stream ":{"type":"http","url":"http://download.ceph.com/qa/ubuntu-12.04.qcow2"}}\' "\'"\' | rbd migration prepare --import-only --source-spec-path - client.0.0\'' [7467574] rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-comp-zlib 4-supported-random-distro$/{centos_latest} 5-data-pool/replicated 6-prepare/qcow2-http 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup conf/{disable-pool-app}} ----------------------------------------------------------------- time: 00:09:58 info: https://pulpito.ceph.com/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/7467574/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/7467574/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1d4ed6cb382f4608944945be4e579cf9 Command failed on smithi183 with status 139: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'echo \'"\'"\'{"type":"qcow","stream ":{"type":"http","url":"http://download.ceph.com/qa/ubuntu-12.04.qcow2"}}\' "\'"\' | rbd migration prepare --import-only --source-spec-path - client.0.0\'' [7467575] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} ----------------------------------------------------------------- time: 04:49:21 info: https://pulpito.ceph.com/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/7467575/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/7467575/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2e6a392c1a9842e19652e6dc9bfccdb7 Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi176 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=33df3bc2b6d55e42c1e87ab8d6f18392df19d216 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object- map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /ho me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror _stress.sh' Pass ================================================================= [7467569] rbd/qemu/{cache/writearound clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/replicated features/defaults msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/qemu_xfstests} time: 01:48:25 info: https://pulpito.ceph.com/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/7467569/ [7467570] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} conf/{disable-pool-app} objectstore/bluestore-comp-lz4 validator/memcheck workloads/c_api_tests_with_journaling} time: 00:54:39 info: https://pulpito.ceph.com/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/7467570/ [7467571] rbd/pwl-cache/tmpfs/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{ubuntu_latest} 4-cache-path 5-cache-mode/ssd 6-cache-size/5G 7-workloads/qemu_xfstests conf/{disable-pool-app}} time: 01:26:44 info: https://pulpito.ceph.com/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/7467571/ [7467573] rbd/maintenance/{base/install clusters/{fixed-3 openstack} conf/{disable-pool-app} objectstore/bluestore-comp-lz4 qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/dynamic_features} time: 02:09:24 info: https://pulpito.ceph.com/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/7467573/ [7467576] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} conf/{disable-pool-app} objectstore/bluestore-low-osd-mem-target validator/memcheck workloads/rbd_mirror} time: 00:54:13 info: https://pulpito.ceph.com/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/7467576/ [7467577] rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-comp-zstd 4-supported-random-distro$/{ubuntu_latest} 5-data-pool/ec 6-prepare/raw-file 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup conf/{disable-pool-app}} time: 02:41:37 info: https://pulpito.ceph.com/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/7467577/ [7467578] rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_latest} workloads/rbd_cli_luks_encryption} time: 00:12:39 info: https://pulpito.ceph.com/dis-2023-11-27_11:31:44-rbd-wip-dis-testing-distro-default-smithi/7467578/