2022-02-15T20:14:03.599 INFO:root:teuthology version: 1.1.0-65c38a8a 2022-02-15T20:14:03.600 INFO:teuthology.results:subset: '60/99' 2022-02-15T20:14:03.600 INFO:teuthology.results:seed: '7292' 2022-02-15T20:31:26.184 INFO:root:teuthology version: 1.1.0-65c38a8a 2022-02-15T20:31:26.214 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-02-16T03:04:55.695 INFO:teuthology.results:Tests finished! gathering results... 2022-02-16T03:04:58.070 INFO:scrape:Found 26 jobs 2022-02-16T03:04:59.724 INFO:scrape:Found 4 distinct failure reasons 2022-02-16T03:04:59.725 INFO:scrape:Dead: b'2022-02-16T02:56:55.761 DEBUG:teuthology.exit:Finished running handlers' 2022-02-16T03:04:59.726 INFO:scrape:3 jobs: ['6685786', '6685775', '6685790'] 2022-02-16T03:04:59.726 INFO:scrape:suites intersection: ['openstack}'] 2022-02-16T03:04:59.727 INFO:scrape:suites union: ['2-cluster/{fix-2', '3-supported-random-distro$/{centos_8}', '4-pool/big-cache', '5-cache-mode/rwl', '6-workloads/qemu_xfstests}', 'clients/mirror', 'cluster/{2-node', 'clusters/{fixed-1', 'features/layering', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'policy/none', 'pool/small-cache-pool', 'rbd-mirror/four-per-cluster', 'rbd/cli/{base/install', 'rbd/mirror-thrash/{base/install', 'rbd/persistent-writeback-cache/{1-base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'workloads/rbd-mirror-fsx-workunit}', 'workloads/rbd_cli_migration}'] 2022-02-16T03:04:59.727 INFO:scrape: 2022-02-16T03:04:59.728 INFO:scrape:Failure: "2022-02-15T20:44:38.676523+0000 mon.a (mon.0) 210 : cluster [WRN] Health check failed: Degraded data redundancy: 1/3400 objects degraded (0.029%), 1 pg degraded (PG_DEGRADED)" in cluster log 2022-02-16T03:04:59.728 INFO:scrape:2 jobs: ['6685793', '6685776'] 2022-02-16T03:04:59.729 INFO:scrape:suites intersection: ['msgr-failures/few', 'openstack}'] 2022-02-16T03:04:59.729 INFO:scrape:suites union: ['clients/mirror', 'cluster/{2-node', 'clusters/{fixed-3', 'config/none', 'min-compat-client/octopus', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'openstack}', 'policy/none', 'pool/ec-data-pool', 'rbd-mirror/four-per-cluster', 'rbd/librbd/{cache/none', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'workloads/c_api_tests_with_journaling}', 'workloads/rbd-mirror-fsx-workunit}'] 2022-02-16T03:04:59.729 INFO:scrape: 2022-02-16T03:04:59.730 INFO:scrape:Failure: Command failed (workunit test rbd/qemu-iotests.sh) on smithi077 with status 13: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=525ff61cfc8516b4d7bed6f819b00a0b6cb7be0a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/qemu-iotests.sh' 2022-02-16T03:04:59.730 INFO:scrape:2 jobs: ['6685789', '6685785'] 2022-02-16T03:04:59.731 INFO:scrape:suites intersection: ['openstack', 'supported-random-distro$/{centos_8}}'] 2022-02-16T03:04:59.731 INFO:scrape:suites union: ['objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'openstack', 'rbd/singleton/{all/qemu-iotests-writearound', 'rbd/singleton/{all/qemu-iotests-writeback', 'supported-random-distro$/{centos_8}}'] 2022-02-16T03:04:59.732 INFO:scrape: 2022-02-16T03:04:59.732 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/cli_generic.sh 2022-02-16T03:04:59.733 INFO:scrape:1 jobs: ['6685777'] 2022-02-16T03:04:59.733 INFO:scrape:suites: ['clusters/{fixed-1', 'features/journaling', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'pool/small-cache-pool', 'rbd/cli/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd_cli_generic}'] 2022-02-16T03:04:59.734 INFO:scrape: 2022-02-16T03:04:59.735 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/ failed: 5 dead: 3 running: 0 waiting: 0 queued: 0 passed: 18 Fail ================================================================= [6685776] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-lz4 pool/ec-data-pool supported-random-distro$/{rhel_8} workloads/c_api_tests_with_journaling} ----------------------------------------------------------------- time: 00:31:03 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685776/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685776/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff598d32c51e49f7a1b9fbc8c60df748 "2022-02-15T20:42:28.811479+0000 mon.a (mon.0) 187 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6685777] rbd/cli/{base/install clusters/{fixed-1 openstack} features/journaling msgr-failures/few objectstore/bluestore-stupid pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_cli_generic} ----------------------------------------------------------------- time: 03:31:11 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685777/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685777/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dd1dc8b3b8d94a32a690407e7cb14c8b Command failed (workunit test rbd/cli_generic.sh) on smithi181 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=525ff61cfc8516b4d7bed6f819b00a0b6cb7be0a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh' [6685785] rbd/singleton/{all/qemu-iotests-writearound objectstore/bluestore-hybrid openstack supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:08:07 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685785/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685785/ sentry: https://sentry.ceph.com/organizations/ceph/?query=696ef99c80f742a98ada35fed055d26a Command failed (workunit test rbd/qemu-iotests.sh) on smithi125 with status 13: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=525ff61cfc8516b4d7bed6f819b00a0b6cb7be0a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/qemu-iotests.sh' [6685789] rbd/singleton/{all/qemu-iotests-writeback objectstore/bluestore-low-osd-mem-target openstack supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:07:48 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685789/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685789/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0603c2437edd4a798a73f77eccef979d Command failed (workunit test rbd/qemu-iotests.sh) on smithi077 with status 13: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=525ff61cfc8516b4d7bed6f819b00a0b6cb7be0a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/qemu-iotests.sh' [6685793] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-fsx-workunit} ----------------------------------------------------------------- time: 00:38:28 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685793/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685793/ "2022-02-15T20:44:38.676523+0000 mon.a (mon.0) 210 : cluster [WRN] Health check failed: Degraded data redundancy: 1/3400 objects degraded (0.029%), 1 pg degraded (PG_DEGRADED)" in cluster log Dead ================================================================= [6685775] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-fsx-workunit} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685775/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685775/ hit max job timeout [6685786] rbd/persistent-writeback-cache/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{centos_8} 4-pool/big-cache 5-cache-mode/rwl 6-workloads/qemu_xfstests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685786/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685786/ hit max job timeout [6685790] rbd/cli/{base/install clusters/{fixed-1 openstack} features/layering msgr-failures/few objectstore/bluestore-comp-snappy pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_cli_migration} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685790/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685790/ hit max job timeout Pass ================================================================= [6685772] rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-stupid 4-supported-random-distro$/{centos_8} 5-pool/replicated-data-pool 6-prepare/qcow2-file 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup} time: 01:33:30 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685772/ [6685773] rbd/qemu/{cache/none clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/ec-cache-pool supported-random-distro$/{centos_8} workloads/qemu_bonnie} time: 00:29:56 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685773/ [6685774] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-stupid qemu/xfstests supported-random-distro$/{centos_8} workloads/dynamic_features_no_cache} time: 02:40:07 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685774/ [6685778] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-bitmap pool/ec-data-pool supported-random-distro$/{rhel_8} workloads/c_api_tests_with_journaling} time: 00:44:42 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685778/ [6685779] rbd/qemu/{cache/none clusters/{fixed-3 openstack} features/readbalance msgr-failures/few objectstore/bluestore-comp-zlib pool/small-cache-pool supported-random-distro$/{centos_8} workloads/qemu_fsstress} time: 00:13:28 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685779/ [6685780] rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-comp-snappy pool/none supported-random-distro$/{centos_8} workloads/rbd_cli_generic} time: 00:19:38 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685780/ [6685781] rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-comp-lz4 4-supported-random-distro$/{centos_8} 5-pool/replicated-data-pool 6-prepare/raw-file 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup} time: 01:20:59 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685781/ [6685782] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-comp-zlib qemu/xfstests supported-random-distro$/{centos_8} workloads/rebuild_object_map} time: 01:29:44 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685782/ [6685783] rbd/qemu/{cache/writethrough clusters/{fixed-3 openstack} features/readbalance msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/none supported-random-distro$/{centos_8} workloads/qemu_fsstress} time: 00:13:56 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685783/ [6685784] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-bitmap pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/python_api_tests} time: 00:18:28 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685784/ [6685787] rbd/qemu/{cache/none clusters/{fixed-3 openstack} features/defaults msgr-failures/few objectstore/bluestore-stupid pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/qemu_xfstests} time: 02:04:17 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685787/ [6685788] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-zstd pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests} time: 00:22:33 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685788/ [6685791] rbd/qemu/{cache/writearound clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_8} workloads/qemu_bonnie} time: 00:23:41 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685791/ [6685792] rbd/cli/{base/install clusters/{fixed-1 openstack} features/defaults msgr-failures/few objectstore/bluestore-comp-zlib pool/ec-data-pool supported-random-distro$/{centos_8} workloads/rbd_cli_generic} time: 00:19:15 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685792/ [6685794] rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-hybrid 4-supported-random-distro$/{centos_8} 5-pool/ec-data-pool 6-prepare/qcow2-file 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup} time: 01:54:25 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685794/ [6685795] rbd/librbd/{cache/writethrough clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-comp-zlib pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_defaults} time: 00:26:41 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685795/ [6685796] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/defaults msgr-failures/few objectstore/bluestore-comp-zstd pool/small-cache-pool supported-random-distro$/{centos_8} workloads/qemu_xfstests} time: 01:03:59 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685796/ [6685797] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} thrashers/cache thrashosds-health workloads/rbd_fsx_nbd} time: 00:26:48 info: http://pulpito.ceph.com/dis-2022-02-15_20:11:52-rbd-wip-dis-testing-distro-default-smithi/6685797/