2020-12-18T11:36:25.654 INFO:root:teuthology version: 1.0.0-8e49dda4 2020-12-18T11:36:25.657 INFO:teuthology.results:subset: '3/10' 2020-12-18T11:36:25.657 INFO:teuthology.results:seed: '8486' 2020-12-18T11:57:45.596 INFO:root:teuthology version: 1.0.0-8e49dda4 2020-12-18T11:57:45.712 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-12-18T13:14:47.118 INFO:teuthology.results:Tests finished! gathering results... 2020-12-18T13:14:51.214 INFO:scrape:Found 35 jobs 2020-12-18T13:14:53.253 INFO:scrape:Found 2 distinct failure reasons 2020-12-18T13:14:53.254 INFO:scrape:Failure: "2020-12-18T12:07:48.069456+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: Degraded data redundancy: 2/2148 objects degraded (0.093%), 1 pg degraded (PG_DEGRADED)" in cluster log 2020-12-18T13:14:53.255 INFO:scrape:1 jobs: ['5718406'] 2020-12-18T13:14:53.255 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-fsx-workunit}'] 2020-12-18T13:14:53.255 INFO:scrape: 2020-12-18T13:14:53.256 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_fsx_compare.sh) on smithi131 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9530fdca3cf9420e89bb02b71e2d65520954eb6b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ARGS=\'\' RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_fsx_compare.sh' 2020-12-18T13:14:53.257 INFO:scrape:1 jobs: ['5718374'] 2020-12-18T13:14:53.257 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd-mirror-fsx-workunit}'] 2020-12-18T13:14:53.257 INFO:scrape: 2020-12-18T13:14:53.259 INFO:teuthology.results:Sending results to to.my.trociny@gmail.com: Test Run: trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 33 Fail ================================================================= [5718374] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-fsx-workunit} ----------------------------------------------------------------- time: 01:04:51 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718374/ log: http://qa-proxy.ceph.com/teuthology/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718374/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0b036f33743141a1b0aff4ee76132c3f Command failed (workunit test rbd/rbd_mirror_fsx_compare.sh) on smithi131 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9530fdca3cf9420e89bb02b71e2d65520954eb6b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ARGS=\'\' RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /ho me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror _fsx_compare.sh' [5718406] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-fsx-workunit} ----------------------------------------------------------------- time: 01:13:19 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718406/ log: http://qa-proxy.ceph.com/teuthology/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718406/ "2020-12-18T12:07:48.069456+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: Degraded data redundancy: 2/2148 objects degraded (0.093%), 1 pg degraded (PG_DEGRADED)" in cluster log Pass ================================================================= [5718373] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-minimum} time: 00:17:49 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718373/ [5718375] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} time: 00:16:11 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718375/ [5718376] rbd/singleton/{all/rbd_mirror objectstore/bluestore-hybrid openstack supported-random-distro$/{centos_8}} time: 00:44:02 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718376/ [5718377] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{rhel_8} workloads/rbd-mirror-workunit-config-key} time: 00:22:59 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718377/ [5718378] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-workunit} time: 00:28:29 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718378/ [5718379] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8} workloads/rbd-mirror-workunit-min-compat-client-octopus} time: 00:23:18 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718379/ [5718380] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} workloads/rbd-mirror-workunit-policy-none} time: 00:22:58 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718380/ [5718381] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-stress-workunit} time: 00:22:46 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718381/ [5718382] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-snappy validator/memcheck workloads/rbd_mirror} time: 00:55:54 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718382/ [5718383] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} workloads/rbd-mirror-workunit-policy-simple} time: 00:23:09 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718383/ [5718384] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} workloads/rbd-mirror-bootstrap-workunit} time: 00:10:48 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718384/ [5718385] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-stress-workunit} time: 00:18:04 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718385/ [5718386] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} workloads/rbd-mirror-ha-workunit} time: 00:17:22 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718386/ [5718387] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-fsx-workunit} time: 01:13:22 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718387/ [5718388] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock} time: 00:17:33 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718388/ [5718389] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-fast-diff} time: 00:17:40 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718389/ [5718390] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-workunit} time: 00:31:08 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718390/ [5718391] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-journaling} time: 00:18:25 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718391/ [5718392] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-minimum} time: 00:15:15 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718392/ [5718393] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-stress-workunit} time: 00:23:35 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718393/ [5718394] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} time: 00:15:50 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718394/ [5718395] rbd/singleton/{all/rbd_mirror objectstore/bluestore-comp-zstd openstack supported-random-distro$/{centos_8}} time: 00:43:33 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718395/ [5718396] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} workloads/rbd-mirror-workunit-config-key} time: 00:24:21 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718396/ [5718397] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-fsx-workunit} time: 01:16:02 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718397/ [5718398] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} workloads/rbd-mirror-workunit-min-compat-client-octopus} time: 00:21:49 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718398/ [5718399] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} workloads/rbd-mirror-workunit-policy-none} time: 00:22:50 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718399/ [5718400] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-workunit} time: 00:34:59 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718400/ [5718401] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-lz4 validator/memcheck workloads/rbd_mirror} time: 00:54:05 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718401/ [5718402] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} workloads/rbd-mirror-workunit-policy-simple} time: 00:23:09 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718402/ [5718403] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} workloads/rbd-mirror-bootstrap-workunit} time: 00:11:22 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718403/ [5718404] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-stress-workunit} time: 00:21:57 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718404/ [5718405] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-ha-workunit} time: 00:14:39 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718405/ [5718407] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock} time: 00:17:26 info: http://pulpito.ceph.com/trociny-2020-12-18_11:04:26-rbd-wip-mgolub-testing-distro-basic-smithi/5718407/