2023-10-10T07:49:58.797 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-10T07:49:58.798 INFO:teuthology.results:subset: '58/99' 2023-10-10T07:49:58.799 INFO:teuthology.results:seed: '6621' 2023-10-10T08:08:14.590 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-10T08:08:14.624 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-10T09:32:21.168 INFO:teuthology.results:Tests finished! gathering results... 2023-10-10T09:32:22.587 INFO:scrape:Found 24 jobs 2023-10-10T09:32:23.363 INFO:scrape:Found 2 distinct failure reasons 2023-10-10T09:32:23.364 INFO:scrape:Failure: "2023-10-10T08:31:38.790298+0000 mon.a (mon.0) 1060 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log 2023-10-10T09:32:23.364 INFO:scrape:2 jobs: ['7419757', '7419759'] 2023-10-10T09:32:23.365 INFO:scrape:suites intersection: ['clusters/{fixed-3', 'conf/{disable-pool-app}', 'data-pool/none', 'extra-conf/none', 'min-compat-client/octopus', 'msgr-failures/few', 'openstack}', 'supported-random-distro$/{ubuntu_latest}'] 2023-10-10T09:32:23.365 INFO:scrape:suites union: ['clusters/{fixed-3', 'conf/{disable-pool-app}', 'data-pool/none', 'extra-conf/none', 'min-compat-client/octopus', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rbd/librbd/{cache/none', 'rbd/librbd/{cache/writeback', 'supported-random-distro$/{ubuntu_latest}', 'workloads/c_api_tests_with_journaling}', 'workloads/c_api_tests}'] 2023-10-10T09:32:23.365 INFO:scrape: 2023-10-10T09:32:23.366 INFO:scrape:Failure: Command failed (workunit test rbd/test_rbd_mirror.sh) on smithi138 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 VALGRIND=\'--tool=memcheck --leak-check=full\' adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_rbd_mirror.sh' 2023-10-10T09:32:23.366 INFO:scrape:1 jobs: ['7419764'] 2023-10-10T09:32:23.367 INFO:scrape:suites: ['centos_latest', 'clusters/{fixed-1', 'conf/{disable-pool-app}', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rbd/valgrind/{base/install', 'validator/memcheck', 'workloads/rbd_mirror}'] 2023-10-10T09:32:23.367 INFO:scrape: 2023-10-10T09:32:23.369 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 21 Fail ================================================================= [7419757] rbd/librbd/{cache/none clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/none extra-conf/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_journaling} ----------------------------------------------------------------- time: 00:34:09 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419757/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419757/ "2023-10-10T08:31:38.790298+0000 mon.a (mon.0) 1060 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log [7419759] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/none extra-conf/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/c_api_tests} ----------------------------------------------------------------- time: 00:49:34 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419759/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419759/ "2023-10-10T08:30:41.825624+0000 mon.a (mon.0) 889 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log [7419764] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} conf/{disable-pool-app} objectstore/bluestore-comp-lz4 validator/memcheck workloads/rbd_mirror} ----------------------------------------------------------------- time: 00:23:04 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419764/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419764/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8d7a61ad3bfe40b4a67dc8131cece34e Command failed (workunit test rbd/test_rbd_mirror.sh) on smithi138 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 VALGRIND=\'--tool=memcheck --leak- check=full\' adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_rbd_mirror.sh' Pass ================================================================= [7419755] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} conf/{disable-pool-app} objectstore/bluestore-comp-zlib validator/memcheck workloads/c_api_tests_with_defaults} time: 01:21:45 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419755/ [7419756] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} conf/{disable-pool-app} objectstore/bluestore-comp-zstd validator/memcheck workloads/c_api_tests_with_journaling} time: 00:48:24 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419756/ [7419758] rbd/immutable-object-cache/{clusters/{fix-2 openstack} conf/{disable-pool-app} pool/ceph_and_immutable_object_cache supported-random-distro$/{centos_latest} workloads/c_api_tests_with_defaults} time: 00:43:38 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419758/ [7419760] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-journaling} time: 00:38:45 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419760/ [7419761] rbd/librbd/{cache/writethrough clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/none extra-conf/none min-compat-client/default msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_latest} workloads/c_api_tests_with_defaults} time: 00:44:35 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419761/ [7419762] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-config-key} time: 00:36:13 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419762/ [7419763] rbd/librbd/{cache/none clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/replicated extra-conf/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_journaling} time: 00:34:16 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419763/ [7419765] rbd/pwl-cache/home/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{centos_latest} 4-cache-path 5-cache-mode/ssd 6-cache-size/1G 7-workloads/c_api_tests_with_defaults conf/{disable-pool-app}} time: 00:45:14 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419765/ [7419766] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} conf/{disable-pool-app} objectstore/bluestore-comp-lz4 validator/memcheck workloads/c_api_tests} time: 01:21:00 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419766/ [7419767] rbd/immutable-object-cache/{clusters/{fix-2 openstack} conf/{disable-pool-app} pool/ceph_and_immutable_object_cache supported-random-distro$/{centos_latest} workloads/qemu_on_immutable_object_cache_and_thrash} time: 01:00:00 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419767/ [7419768] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} time: 00:49:23 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419768/ [7419769] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} conf/{disable-pool-app} objectstore/bluestore-comp-snappy validator/memcheck workloads/c_api_tests_with_defaults} time: 01:21:50 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419769/ [7419770] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/replicated extra-conf/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_latest} workloads/python_api_tests} time: 00:20:28 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419770/ [7419771] rbd/librbd/{cache/writethrough clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/ec extra-conf/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_latest} workloads/python_api_tests_with_defaults} time: 00:20:26 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419771/ [7419772] rbd/nbd/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/rbd_nbd} time: 00:12:22 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419772/ [7419773] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/ec extra-conf/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_latest} workloads/c_api_tests} time: 00:42:23 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419773/ [7419774] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} conf/{disable-pool-app} objectstore/bluestore-comp-zlib validator/memcheck workloads/c_api_tests_with_journaling} time: 00:50:15 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419774/ [7419775] rbd/librbd/{cache/writethrough clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/none extra-conf/none min-compat-client/default msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_latest} workloads/c_api_tests_with_defaults} time: 00:45:40 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419775/ [7419776] rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-comp-zstd 4-supported-random-distro$/{centos_latest} 5-data-pool/replicated 6-prepare/qcow2-http 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup conf/{disable-pool-app}} time: 01:03:11 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419776/ [7419777] rbd/cli_v1/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/format-1 msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_latest} workloads/rbd_cli_generic} time: 00:28:18 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419777/ [7419778] rbd/librbd/{cache/none clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/replicated extra-conf/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_journaling} time: 00:34:21 info: http://pulpito.ceph.com/dis-2023-10-10_07:49:16-rbd-wip-dis-testing-distro-default-smithi/7419778/