2022-04-19T09:00:50.163 INFO:root:teuthology version: 1.1.0-b21f9558 2022-04-19T09:00:50.164 INFO:teuthology.results:subset: '3/4' 2022-04-19T09:00:50.164 INFO:teuthology.results:seed: '2849' 2022-04-19T09:27:23.207 INFO:root:teuthology version: 1.1.0-b21f9558 2022-04-19T09:27:23.218 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-19T10:39:28.456 INFO:teuthology.results:Tests finished! gathering results... 2022-04-19T10:39:29.732 INFO:scrape:Found 18 jobs 2022-04-19T10:39:30.247 INFO:scrape:Found 1 distinct failure reasons 2022-04-19T10:39:30.248 INFO:scrape:Failure: "2022-04-19T09:19:16.212702+0000 mon.a (mon.0) 396 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2022-04-19T10:39:30.248 INFO:scrape:1 jobs: ['6796884'] 2022-04-19T10:39:30.248 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs:volumes/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/misc}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-19T10:39:30.248 INFO:scrape: 2022-04-19T10:39:30.249 INFO:teuthology.results:Sending results to nshelke@redhat.com: Test Run: nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/ failed: 1 dead: 0 running: 0 waiting: 0 queued: 0 passed: 17 Fail ================================================================= [6796884] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} ----------------------------------------------------------------- time: 00:17:08 info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796884/ log: http://qa-proxy.ceph.com/teuthology/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796884/ "2022-04-19T09:19:16.212702+0000 mon.a (mon.0) 396 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log Pass ================================================================= [6796885] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} time: 00:17:33 info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796885/ [6796886] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} time: 00:17:14 info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796886/ [6796887] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}} time: 00:21:32 info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796887/ [6796888] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}} time: 00:20:54 info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796888/ [6796889] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}} time: 00:21:07 info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796889/ [6796890] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} time: 01:02:42 info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796890/ [6796891] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} time: 01:03:52 info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796891/ [6796892] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} time: 01:03:06 info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796892/ [6796893] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} time: 00:41:11 info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796893/ [6796894] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} time: 00:40:43 info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796894/ [6796895] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} time: 00:40:57 info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796895/ [6796896] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} time: 00:16:34 info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796896/ [6796897] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} time: 00:17:08 info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796897/ [6796898] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} time: 00:17:01 info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796898/ [6796899] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}} time: 00:22:20 info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796899/ [6796900] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}} time: 00:22:55 info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796900/ [6796901] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}} time: 00:22:43 info: http://pulpito.ceph.com/nshelke-2022-04-19_08:32:41-fs:volumes-wip-nshelke-subvolume-metadata-mgmt-distro-default-smithi/6796901/