2021-08-03T09:46:55.172 INFO:root:teuthology version: 1.1.0-73aa7e3b 2021-08-03T09:46:55.174 INFO:teuthology.results:subset: '0/10' 2021-08-03T09:46:55.174 INFO:teuthology.results:seed: '9941' 2021-08-03T10:01:07.024 INFO:root:teuthology version: 1.1.0-73aa7e3b 2021-08-03T10:01:07.028 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-08-03T11:23:13.023 INFO:teuthology.results:Tests finished! gathering results... 2021-08-03T11:23:15.943 INFO:scrape:Found 30 jobs 2021-08-03T11:23:17.098 INFO:scrape:Found 3 distinct failure reasons 2021-08-03T11:23:17.099 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2021-08-03T11:23:17.099 INFO:scrape:2 jobs: ['6309239', '6309242'] 2021-08-03T11:23:17.100 INFO:scrape:suites intersection: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'openstack}', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install'] 2021-08-03T11:23:17.100 INFO:scrape:suites union: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'policy/none', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'workloads/rbd-mirror-journal-stress-workunit}', 'workloads/rbd-mirror-journal-workunit}'] 2021-08-03T11:23:17.101 INFO:scrape: 2021-08-03T11:23:17.101 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi183 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d23e2d9cb3e51924b0b8761a4f348d92a28b740f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' 2021-08-03T11:23:17.101 INFO:scrape:2 jobs: ['6309241', '6309225'] 2021-08-03T11:23:17.102 INFO:scrape:suites intersection: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'openstack}', 'rbd/mirror/{base/install'] 2021-08-03T11:23:17.102 INFO:scrape:suites union: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff}', 'workloads/rbd-mirror-snapshot-workunit-minimum}'] 2021-08-03T11:23:17.103 INFO:scrape: 2021-08-03T11:23:17.103 INFO:scrape:Failure: "2021-08-03T10:18:35.472366+0000 mon.a (mon.0) 110 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log 2021-08-03T11:23:17.104 INFO:scrape:1 jobs: ['6309246'] 2021-08-03T11:23:17.104 INFO:scrape:suites: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd-mirror-stress-workunit-min-compat-client-octopus}'] 2021-08-03T11:23:17.105 INFO:scrape: 2021-08-03T11:23:17.107 INFO:teuthology.results:Sending results to to.my.trociny@gmail.com: Test Run: trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 25 Fail ================================================================= [6309225] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-workunit-minimum} ----------------------------------------------------------------- time: 00:16:37 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309225/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309225/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a77dd3a29f81442baefa3323a650405d Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi125 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d23e2d9cb3e51924b0b8761a4f348d92a28b740f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6309239] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-stress-workunit} ----------------------------------------------------------------- time: 00:18:30 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309239/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309239/ sentry: https://sentry.ceph.com/organizations/ceph/?query=60da605e905947798ada900c208f0571 reached maximum tries (90) after waiting for 540 seconds [6309241] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-fast-diff} ----------------------------------------------------------------- time: 00:14:35 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309241/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309241/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f276bceaeaeb4c5ea00078077bb20cf4 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi183 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d23e2d9cb3e51924b0b8761a4f348d92a28b740f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6309242] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-workunit} ----------------------------------------------------------------- time: 00:27:54 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309242/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309242/ sentry: https://sentry.ceph.com/organizations/ceph/?query=06e9759d39dc46738394c3875c49e98a reached maximum tries (90) after waiting for 540 seconds [6309246] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_8} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} ----------------------------------------------------------------- time: 00:18:30 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309246/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309246/ sentry: https://sentry.ceph.com/organizations/ceph/?query=acf2a8d25d684dc99c93bf19573fa774 "2021-08-03T10:18:35.472366+0000 mon.a (mon.0) 110 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log Pass ================================================================= [6309217] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-bootstrap-workunit} time: 00:09:48 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309217/ [6309218] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-fsx-workunit} time: 00:47:00 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309218/ [6309219] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} workloads/rbd-mirror-ha-workunit} time: 00:17:24 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309219/ [6309220] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-stress-workunit} time: 01:00:26 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309220/ [6309221] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-workunit-exclusive-lock} time: 00:23:55 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309221/ [6309222] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-fast-diff} time: 00:20:55 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309222/ [6309223] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-workunit} time: 00:32:46 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309223/ [6309224] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-journaling} time: 00:31:17 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309224/ [6309226] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} time: 00:50:08 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309226/ [6309227] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} time: 00:52:42 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309227/ [6309228] rbd/singleton/{all/rbd_mirror objectstore/bluestore-comp-lz4 openstack supported-random-distro$/{ubuntu_latest}} time: 00:41:16 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309228/ [6309229] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-config-key} time: 00:22:54 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309229/ [6309230] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} time: 00:52:08 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309230/ [6309231] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_8} workloads/rbd-mirror-workunit-min-compat-client-octopus} time: 00:24:32 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309231/ [6309232] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8} workloads/rbd-mirror-workunit-policy-none} time: 00:24:22 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309232/ [6309233] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} time: 00:50:05 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309233/ [6309234] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-stupid validator/memcheck workloads/rbd_mirror} time: 00:58:40 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309234/ [6309235] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} workloads/rbd-mirror-workunit-policy-simple} time: 00:24:09 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309235/ [6309236] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} workloads/rbd-mirror-bootstrap-workunit} time: 00:12:03 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309236/ [6309237] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-fsx-workunit} time: 00:44:57 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309237/ [6309238] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} workloads/rbd-mirror-ha-workunit} time: 00:26:20 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309238/ [6309240] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock} time: 00:23:32 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309240/ [6309243] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-journaling} time: 00:22:43 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309243/ [6309244] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-minimum} time: 00:31:09 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309244/ [6309245] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} time: 01:10:20 info: http://pulpito.ceph.com/trociny-2021-08-03_09:45:36-rbd-wip-mgolub-testing-distro-basic-smithi/6309245/