2021-08-01T06:10:35.927 INFO:root:teuthology version: 1.1.0-73aa7e3b 2021-08-01T06:10:35.928 INFO:teuthology.results:subset: '1/10' 2021-08-01T06:10:35.929 INFO:teuthology.results:seed: '3384' 2021-08-01T06:23:09.495 INFO:root:teuthology version: 1.1.0-73aa7e3b 2021-08-01T06:23:09.528 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-08-01T07:26:14.253 INFO:teuthology.results:Tests finished! gathering results... 2021-08-01T07:26:16.510 INFO:scrape:Found 30 jobs 2021-08-01T07:26:18.619 INFO:scrape:Found 2 distinct failure reasons 2021-08-01T07:26:18.620 INFO:scrape:Failure: "2021-08-01T06:37:37.319255+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: Degraded data redundancy: 2 pgs degraded (PG_DEGRADED)" in cluster log 2021-08-01T07:26:18.621 INFO:scrape:2 jobs: ['6305998', '6306008'] 2021-08-01T07:26:18.621 INFO:scrape:suites intersection: ['cluster/{2-node', 'msgr-failures/few', 'openstack}'] 2021-08-01T07:26:18.622 INFO:scrape:suites union: ['clients/mirror', 'clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-fsx-workunit}', 'workloads/rbd-mirror-workunit-config-key}'] 2021-08-01T07:26:18.622 INFO:scrape: 2021-08-01T07:26:18.623 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2021-08-01T07:26:18.623 INFO:scrape:4 jobs: ['6306009', '6305993', '6306002', '6305987'] 2021-08-01T07:26:18.624 INFO:scrape:suites intersection: ['cluster/{2-node', 'msgr-failures/few', 'openstack}'] 2021-08-01T07:26:18.624 INFO:scrape:suites union: ['clients/mirror', 'clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'objectstore/filestore-xfs', 'openstack}', 'policy/none', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-fsx-workunit}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'workloads/rbd-mirror-snapshot-workunit-exclusive-lock}', 'workloads/rbd-mirror-snapshot-workunit-journaling}'] 2021-08-01T07:26:18.625 INFO:scrape: 2021-08-01T07:26:18.627 INFO:teuthology.results:Sending results to to.my.trociny@gmail.com: Test Run: trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/ failed: 6 dead: 0 running: 0 waiting: 0 queued: 0 passed: 24 Fail ================================================================= [6305987] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-fsx-workunit} ----------------------------------------------------------------- time: 00:53:02 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305987/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305987/ reached maximum tries (50) after waiting for 300 seconds [6305993] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-journaling} ----------------------------------------------------------------- time: 00:17:03 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305993/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305993/ sentry: https://sentry.ceph.com/organizations/ceph/?query=271551b37a8c4d4c869b55dd37ace440 reached maximum tries (90) after waiting for 540 seconds [6305998] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-config-key} ----------------------------------------------------------------- time: 00:22:07 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305998/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305998/ "2021-08-01T06:37:37.319255+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: Degraded data redundancy: 2 pgs degraded (PG_DEGRADED)" in cluster log [6306002] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-stress-workunit-minimum} ----------------------------------------------------------------- time: 00:19:19 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6306002/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6306002/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4313cac7ece748fca1f8faf6c72159b5 reached maximum tries (90) after waiting for 540 seconds [6306008] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-fsx-workunit} ----------------------------------------------------------------- time: 00:32:50 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6306008/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6306008/ "2021-08-01T06:41:24.787222+0000 mon.a (mon.0) 212 : cluster [WRN] Health check failed: Degraded data redundancy: 5/1070 objects degraded (0.467%), 1 pg degraded (PG_DEGRADED)" in cluster log [6306009] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-exclusive-lock} ----------------------------------------------------------------- time: 00:16:09 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6306009/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6306009/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5f9f1fff41dd4dfc955344e208ab23b6 reached maximum tries (90) after waiting for 540 seconds Pass ================================================================= [6305982] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} workloads/rbd-mirror-workunit-policy-none} time: 00:23:36 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305982/ [6305983] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-minimum} time: 00:58:47 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305983/ [6305984] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-low-osd-mem-target validator/memcheck workloads/rbd_mirror} time: 00:55:58 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305984/ [6305985] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-workunit-policy-simple} time: 00:24:03 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305985/ [6305986] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-bootstrap-workunit} time: 00:09:38 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305986/ [6305988] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} workloads/rbd-mirror-ha-workunit} time: 00:24:56 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305988/ [6305989] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-stress-workunit} time: 01:04:09 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305989/ [6305990] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock} time: 00:30:48 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305990/ [6305991] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-fast-diff} time: 00:31:15 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305991/ [6305992] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-workunit} time: 00:40:36 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305992/ [6305994] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-workunit-minimum} time: 00:22:36 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305994/ [6305995] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} time: 00:51:14 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305995/ [6305996] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} time: 00:53:26 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305996/ [6305997] rbd/singleton/{all/rbd_mirror objectstore/filestore-xfs openstack supported-random-distro$/{rhel_8}} time: 00:51:44 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305997/ [6305999] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} time: 00:52:13 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6305999/ [6306000] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} workloads/rbd-mirror-workunit-min-compat-client-octopus} time: 00:31:18 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6306000/ [6306001] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} workloads/rbd-mirror-workunit-policy-none} time: 00:32:26 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6306001/ [6306003] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-hybrid validator/memcheck workloads/rbd_mirror} time: 00:55:41 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6306003/ [6306004] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} workloads/rbd-mirror-workunit-policy-simple} time: 00:23:54 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6306004/ [6306005] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} workloads/rbd-mirror-bootstrap-workunit} time: 00:12:12 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6306005/ [6306006] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} time: 00:48:01 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6306006/ [6306007] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{rhel_8} workloads/rbd-mirror-ha-workunit} time: 00:24:33 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6306007/ [6306010] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-fast-diff} time: 00:20:54 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6306010/ [6306011] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-journal-stress-workunit} time: 00:54:25 info: http://pulpito.ceph.com/trociny-2021-08-01_06:09:53-rbd-wip-mgolub-testing-distro-basic-smithi/6306011/