2022-03-12T18:18:58.514 INFO:root:teuthology version: 1.1.0-fbfc7b38 2022-03-12T18:18:58.515 INFO:teuthology.results:subset: '74/99' 2022-03-12T18:18:58.516 INFO:teuthology.results:seed: '2678' 2022-03-12T18:26:04.394 INFO:root:teuthology version: 1.1.0-fbfc7b38 2022-03-12T18:26:04.400 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-03-13T01:00:34.032 INFO:teuthology.results:Tests finished! gathering results... 2022-03-13T01:00:36.134 INFO:scrape:Found 12 jobs 2022-03-13T01:00:51.184 INFO:scrape:Found 3 distinct failure reasons 2022-03-13T01:00:51.185 INFO:scrape:Failure: Command failed (workunit test rbd/cli_generic.sh) on smithi152 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0bcb5cf12d0ee89abf1668d1e0d3b69e71ef3e78 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh' 2022-03-13T01:00:51.186 INFO:scrape:2 jobs: ['6733938', '6733944'] 2022-03-13T01:00:51.187 INFO:scrape:suites intersection: ['clusters/{fixed-1', 'msgr-failures/few', 'openstack}', 'rbd/cli/{base/install', 'workloads/rbd_cli_generic}'] 2022-03-13T01:00:51.187 INFO:scrape:suites union: ['clusters/{fixed-1', 'features/journaling', 'features/layering', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-hybrid', 'openstack}', 'pool/ec-data-pool', 'pool/replicated-data-pool', 'rbd/cli/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd_cli_generic}'] 2022-03-13T01:00:51.188 INFO:scrape: 2022-03-13T01:00:51.188 INFO:scrape:Failure: "2022-03-12T18:48:36.087492+0000 mon.a (mon.0) 786 : cluster [WRN] Health check failed: Degraded data redundancy: 1/214 objects degraded (0.467%), 1 pg degraded (PG_DEGRADED)" in cluster log 2022-03-13T01:00:51.189 INFO:scrape:1 jobs: ['6733942'] 2022-03-13T01:00:51.189 INFO:scrape:suites: ['clusters/{fixed-3', 'config/copy-on-read', 'min-compat-client/octopus', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'openstack}', 'pool/small-cache-pool', 'rbd/librbd/{cache/writearound', 'supported-random-distro$/{ubuntu_latest}', 'workloads/c_api_tests_with_journaling}'] 2022-03-13T01:00:51.190 INFO:scrape: 2022-03-13T01:00:51.190 INFO:scrape:Dead: b'2022-03-13T00:55:43.148 DEBUG:teuthology.exit:Finished running handlers' 2022-03-13T01:00:51.191 INFO:scrape:1 jobs: ['6733939'] 2022-03-13T01:00:51.191 INFO:scrape:suites: ['2-cluster/{fix-2', '3-supported-random-distro$/{rhel_8}', '4-pool/cache', '5-cache-mode/rwl', '6-workloads/qemu_xfstests}', 'openstack}', 'rbd/persistent-writeback-cache/{1-base/install'] 2022-03-13T01:00:51.191 INFO:scrape: 2022-03-13T01:00:51.192 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi/ failed: 3 dead: 1 running: 0 waiting: 0 queued: 0 passed: 8 Fail ================================================================= [6733938] rbd/cli/{base/install clusters/{fixed-1 openstack} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/ec-data-pool supported-random-distro$/{centos_8} workloads/rbd_cli_generic} ----------------------------------------------------------------- time: 00:17:01 info: http://pulpito.ceph.com/dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi/6733938/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi/6733938/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1a3113ab46994f24833685b6a86befda Command failed (workunit test rbd/cli_generic.sh) on smithi152 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0bcb5cf12d0ee89abf1668d1e0d3b69e71ef3e78 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh' [6733942] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-zstd pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_journaling} ----------------------------------------------------------------- time: 00:32:52 info: http://pulpito.ceph.com/dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi/6733942/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi/6733942/ "2022-03-12T18:48:36.087492+0000 mon.a (mon.0) 786 : cluster [WRN] Health check failed: Degraded data redundancy: 1/214 objects degraded (0.467%), 1 pg degraded (PG_DEGRADED)" in cluster log [6733944] rbd/cli/{base/install clusters/{fixed-1 openstack} features/journaling msgr-failures/few objectstore/bluestore-comp-lz4 pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_generic} ----------------------------------------------------------------- time: 00:16:07 info: http://pulpito.ceph.com/dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi/6733944/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi/6733944/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2a7ee73bd6444b01aff167c0a0966bfa Command failed (workunit test rbd/cli_generic.sh) on smithi141 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0bcb5cf12d0ee89abf1668d1e0d3b69e71ef3e78 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh' Dead ================================================================= [6733939] rbd/persistent-writeback-cache/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{rhel_8} 4-pool/cache 5-cache-mode/rwl 6-workloads/qemu_xfstests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi/6733939/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi/6733939/ hit max job timeout Pass ================================================================= [6733940] rbd/librbd/{cache/writethrough clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-zlib pool/none supported-random-distro$/{rhel_8} workloads/c_api_tests} time: 00:37:36 info: http://pulpito.ceph.com/dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi/6733940/ [6733941] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-hybrid pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_journaling} time: 00:32:19 info: http://pulpito.ceph.com/dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi/6733941/ [6733943] rbd/cli/{base/install clusters/{fixed-1 openstack} features/defaults msgr-failures/few objectstore/bluestore-bitmap pool/none supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_migration} time: 00:26:02 info: http://pulpito.ceph.com/dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi/6733943/ [6733945] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-fsx-workunit} time: 00:38:24 info: http://pulpito.ceph.com/dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi/6733945/ [6733946] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-zstd pool/ec-data-pool supported-random-distro$/{centos_8} workloads/python_api_tests} time: 00:20:12 info: http://pulpito.ceph.com/dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi/6733946/ [6733947] rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_deep_copy} time: 00:15:37 info: http://pulpito.ceph.com/dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi/6733947/ [6733948] rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-comp-zlib pool/none supported-random-distro$/{rhel_8} workloads/rbd_cli_generic} time: 00:31:25 info: http://pulpito.ceph.com/dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi/6733948/ [6733949] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} time: 00:45:46 info: http://pulpito.ceph.com/dis-2022-03-12_18:17:39-rbd-wip-dis-testing-distro-default-smithi/6733949/