2021-08-01T10:09:45.104 INFO:root:teuthology version: 1.1.0-73aa7e3b 2021-08-01T10:09:45.107 INFO:teuthology.results:subset: '0/10' 2021-08-01T10:09:45.107 INFO:teuthology.results:seed: '6416' 2021-08-01T10:10:11.035 INFO:root:teuthology version: 1.1.0-73aa7e3b 2021-08-01T10:10:11.036 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-08-01T11:22:16.158 INFO:teuthology.results:Tests finished! gathering results... 2021-08-01T11:22:19.212 INFO:scrape:Found 30 jobs 2021-08-01T11:22:21.266 INFO:scrape:Found 3 distinct failure reasons 2021-08-01T11:22:21.267 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2021-08-01T11:22:21.268 INFO:scrape:2 jobs: ['6306482', '6306471'] 2021-08-01T11:22:21.268 INFO:scrape:suites intersection: ['cluster/{2-node', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}'] 2021-08-01T11:22:21.269 INFO:scrape:suites union: ['clients/mirror', 'clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-journal-workunit}', 'workloads/rbd-mirror-workunit-min-compat-client-octopus}'] 2021-08-01T11:22:21.269 INFO:scrape: 2021-08-01T11:22:21.269 INFO:scrape:Failure: "2021-08-01T10:32:40.459511+0000 mon.a (mon.0) 102 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log 2021-08-01T11:22:21.270 INFO:scrape:2 jobs: ['6306462', '6306458'] 2021-08-01T11:22:21.270 INFO:scrape:suites intersection: ['cluster/{2-node', 'msgr-failures/few', 'openstack}'] 2021-08-01T11:22:21.271 INFO:scrape:suites union: ['clients/mirror', 'clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zlib', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'rbd/mirror/{base/install', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-fsx-workunit}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff}'] 2021-08-01T11:22:21.271 INFO:scrape: 2021-08-01T11:22:21.272 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi093 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8105076f60f36b663cb09a6c46f63140b873dfe6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' 2021-08-01T11:22:21.272 INFO:scrape:2 jobs: ['6306465', '6306483'] 2021-08-01T11:22:21.273 INFO:scrape:suites intersection: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'openstack}', 'rbd/mirror/{base/install'] 2021-08-01T11:22:21.273 INFO:scrape:suites union: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-workunit-journaling}', 'workloads/rbd-mirror-snapshot-workunit-minimum}'] 2021-08-01T11:22:21.273 INFO:scrape: 2021-08-01T11:22:21.276 INFO:teuthology.results:Sending results to to.my.trociny@gmail.com: Test Run: trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/ failed: 6 dead: 0 running: 0 waiting: 0 queued: 0 passed: 24 Fail ================================================================= [6306458] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-fsx-workunit} ----------------------------------------------------------------- time: 00:42:53 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306458/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306458/ "2021-08-01T10:29:41.300220+0000 mon.a (mon.0) 267 : cluster [WRN] Health check failed: Degraded data redundancy: 10/9290 objects degraded (0.108%), 2 pgs degraded (PG_DEGRADED)" in cluster log [6306462] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-fast-diff} ----------------------------------------------------------------- time: 00:26:50 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306462/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306462/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6f755b56577d47659f94576934cad44a "2021-08-01T10:32:40.459511+0000 mon.a (mon.0) 102 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6306465] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-minimum} ----------------------------------------------------------------- time: 00:17:55 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306465/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306465/ sentry: https://sentry.ceph.com/organizations/ceph/?query=99d544a149f54ebeb48cc805ab3df026 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi093 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8105076f60f36b663cb09a6c46f63140b873dfe6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6306471] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-min-compat-client-octopus} ----------------------------------------------------------------- time: 00:17:13 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306471/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306471/ sentry: https://sentry.ceph.com/organizations/ceph/?query=79347bea04634a2fa23947a382ab25b1 reached maximum tries (90) after waiting for 540 seconds [6306482] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-journal-workunit} ----------------------------------------------------------------- time: 00:21:22 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306482/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306482/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d40af6bf0fdd4b36add97f6ac7849ad2 reached maximum tries (90) after waiting for 540 seconds [6306483] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-workunit-journaling} ----------------------------------------------------------------- time: 00:18:26 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306483/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306483/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5518bb54c9da47f382965bb602773930 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi039 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8105076f60f36b663cb09a6c46f63140b873dfe6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,journaling RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' Pass ================================================================= [6306457] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-bootstrap-workunit} time: 00:13:43 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306457/ [6306459] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} workloads/rbd-mirror-ha-workunit} time: 00:18:46 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306459/ [6306460] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-stress-workunit} time: 01:06:24 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306460/ [6306461] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock} time: 00:24:26 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306461/ [6306463] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-workunit} time: 00:41:03 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306463/ [6306464] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-workunit-journaling} time: 00:23:58 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306464/ [6306466] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} time: 00:51:48 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306466/ [6306467] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} time: 01:01:09 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306467/ [6306468] rbd/singleton/{all/rbd_mirror objectstore/bluestore-comp-lz4 openstack supported-random-distro$/{centos_8}} time: 00:44:58 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306468/ [6306469] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{rhel_8} workloads/rbd-mirror-workunit-config-key} time: 00:32:50 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306469/ [6306470] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} time: 00:53:48 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306470/ [6306472] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} workloads/rbd-mirror-workunit-policy-none} time: 00:33:11 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306472/ [6306473] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-minimum} time: 00:51:23 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306473/ [6306474] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-stupid validator/memcheck workloads/rbd_mirror} time: 00:59:02 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306474/ [6306475] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} workloads/rbd-mirror-workunit-policy-simple} time: 00:33:31 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306475/ [6306476] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} workloads/rbd-mirror-bootstrap-workunit} time: 00:20:35 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306476/ [6306477] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-fsx-workunit} time: 00:42:02 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306477/ [6306478] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} workloads/rbd-mirror-ha-workunit} time: 00:18:56 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306478/ [6306479] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-stress-workunit} time: 01:06:01 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306479/ [6306480] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock} time: 00:25:31 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306480/ [6306481] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-fast-diff} time: 00:26:07 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306481/ [6306484] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-minimum} time: 00:32:02 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306484/ [6306485] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} time: 00:51:25 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306485/ [6306486] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} time: 00:53:29 info: http://pulpito.ceph.com/trociny-2021-08-01_10:09:28-rbd-wip-mgolub-testing-distro-basic-smithi/6306486/