2022-02-22T18:00:39.123 INFO:root:teuthology version: 1.1.0-eea25212 2022-02-22T18:00:39.124 INFO:teuthology.results:subset: '77/99' 2022-02-22T18:00:39.125 INFO:teuthology.results:seed: '2874' 2022-02-22T18:01:27.252 INFO:root:teuthology version: 1.1.0-eea25212 2022-02-22T18:01:27.258 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-02-23T00:53:56.983 INFO:teuthology.results:Tests finished! gathering results... 2022-02-23T00:53:58.441 INFO:scrape:Found 14 jobs 2022-02-23T00:53:59.400 INFO:scrape:Found 4 distinct failure reasons 2022-02-23T00:53:59.401 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi164 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e7068e1f7439039cd93008a9268fbb7674fbeb4e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2022-02-23T00:53:59.402 INFO:scrape:1 jobs: ['6700769'] 2022-02-23T00:53:59.403 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}'] 2022-02-23T00:53:59.403 INFO:scrape: 2022-02-23T00:53:59.404 INFO:scrape:Failure: "2022-02-22T18:39:12.926268+0000 mon.a (mon.0) 735 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log 2022-02-23T00:53:59.405 INFO:scrape:1 jobs: ['6700761'] 2022-02-23T00:53:59.405 INFO:scrape:suites: ['clusters/{fixed-3', 'config/none', 'min-compat-client/octopus', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'openstack}', 'pool/ec-data-pool', 'rbd/librbd/{cache/none', 'supported-random-distro$/{rhel_8}', 'workloads/c_api_tests_with_journaling}'] 2022-02-23T00:53:59.406 INFO:scrape: 2022-02-23T00:53:59.406 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/qemu_dynamic_features.sh 2022-02-23T00:53:59.407 INFO:scrape:2 jobs: ['6700767', '6700765'] 2022-02-23T00:53:59.407 INFO:scrape:suites intersection: ['clusters/{fixed-3', 'openstack}', 'qemu/xfstests', 'rbd/maintenance/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/dynamic_features_no_cache}'] 2022-02-23T00:53:59.408 INFO:scrape:suites union: ['clusters/{fixed-3', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zstd', 'openstack}', 'qemu/xfstests', 'rbd/maintenance/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/dynamic_features_no_cache}'] 2022-02-23T00:53:59.408 INFO:scrape: 2022-02-23T00:53:59.408 INFO:scrape:Dead: b'2022-02-23T00:38:39.391 DEBUG:teuthology.exit:Finished running handlers' 2022-02-23T00:53:59.409 INFO:scrape:2 jobs: ['6700758', '6700760'] 2022-02-23T00:53:59.409 INFO:scrape:suites intersection: ['openstack}'] 2022-02-23T00:53:59.410 INFO:scrape:suites union: ['2-cluster/{fix-2', '3-supported-random-distro$/{ubuntu_latest}', '4-pool/cache', '5-cache-mode/rwl', '6-workloads/qemu_xfstests}', 'clusters/{fixed-3', 'features/journaling', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'openstack}', 'pool/ec-cache-pool', 'rbd/persistent-writeback-cache/{1-base/install', 'rbd/qemu/{cache/none', 'supported-random-distro$/{ubuntu_latest}', 'workloads/qemu_xfstests}'] 2022-02-23T00:53:59.410 INFO:scrape: 2022-02-23T00:53:59.411 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/ failed: 4 dead: 2 running: 0 waiting: 0 queued: 0 passed: 8 Fail ================================================================= [6700761] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-hybrid pool/ec-data-pool supported-random-distro$/{rhel_8} workloads/c_api_tests_with_journaling} ----------------------------------------------------------------- time: 00:44:41 info: http://pulpito.ceph.com/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700761/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700761/ "2022-02-22T18:39:12.926268+0000 mon.a (mon.0) 735 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log [6700765] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-comp-lz4 qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/dynamic_features_no_cache} ----------------------------------------------------------------- time: 03:16:34 info: http://pulpito.ceph.com/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700765/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700765/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3f0712b2a45c4d4fbae2b8dec7586676 Command failed (workunit test rbd/qemu_dynamic_features.sh) on smithi092 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e7068e1f7439039cd93008a9268fbb7674fbeb4e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 IMAGE_NAME=client.0.1-clone adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /ho me/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/qemu_dynamic_features.sh ' [6700767] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-comp-zstd qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/dynamic_features_no_cache} ----------------------------------------------------------------- time: 03:18:14 info: http://pulpito.ceph.com/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700767/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700767/ sentry: https://sentry.ceph.com/organizations/ceph/?query=62ed4c1ff86648a18f2418a7cdf3f854 Command failed (workunit test rbd/qemu_dynamic_features.sh) on smithi163 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e7068e1f7439039cd93008a9268fbb7674fbeb4e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 IMAGE_NAME=client.0.1-clone adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /ho me/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/qemu_dynamic_features.sh ' [6700769] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-minimum} ----------------------------------------------------------------- time: 00:19:40 info: http://pulpito.ceph.com/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700769/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700769/ sentry: https://sentry.ceph.com/organizations/ceph/?query=df225d4916ac4381b99663c7af49b732 Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi164 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e7068e1f7439039cd93008a9268fbb7674fbeb4e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /ho me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror _stress.sh' Dead ================================================================= [6700758] rbd/persistent-writeback-cache/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{ubuntu_latest} 4-pool/cache 5-cache-mode/rwl 6-workloads/qemu_xfstests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700758/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700758/ hit max job timeout [6700760] rbd/qemu/{cache/none clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/bluestore-bitmap pool/ec-cache-pool supported-random-distro$/{ubuntu_latest} workloads/qemu_xfstests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700760/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700760/ hit max job timeout Pass ================================================================= [6700757] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-comp-zstd qemu/xfstests supported-random-distro$/{rhel_8} workloads/rebuild_object_map} time: 01:42:53 info: http://pulpito.ceph.com/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700757/ [6700759] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-zlib pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests} time: 00:27:36 info: http://pulpito.ceph.com/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700759/ [6700762] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/none supported-random-distro$/{ubuntu_latest} workloads/fsx} time: 00:18:15 info: http://pulpito.ceph.com/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700762/ [6700763] rbd/basic/{base/install cachepool/small clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/rbd_lock_and_fence} time: 00:10:25 info: http://pulpito.ceph.com/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700763/ [6700764] rbd/qemu/{cache/writethrough clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/bluestore-comp-zlib pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/qemu_xfstests} time: 04:37:58 info: http://pulpito.ceph.com/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700764/ [6700766] rbd/librbd/{cache/writethrough clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-comp-lz4 pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/c_api_tests_with_defaults} time: 00:29:55 info: http://pulpito.ceph.com/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700766/ [6700768] rbd/librbd/{cache/writethrough clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-comp-zstd pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/python_api_tests_with_defaults} time: 00:25:12 info: http://pulpito.ceph.com/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700768/ [6700770] rbd/librbd/{cache/writethrough clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-stupid pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_defaults} time: 00:29:56 info: http://pulpito.ceph.com/dis-2022-02-22_16:30:06-rbd-wip-dis-testing-distro-default-smithi/6700770/