2022-05-30T19:44:21.461 INFO:root:teuthology version: 1.1.1.dev348+g8df7ff2b 2022-05-30T19:44:21.462 INFO:teuthology.results:subset: '1566/4096' 2022-05-30T19:44:21.462 INFO:teuthology.results:seed: '2140' 2022-05-30T20:22:20.168 INFO:root:teuthology version: 1.1.1.dev348+g8df7ff2b 2022-05-30T20:22:20.177 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-05-30T21:46:26.823 INFO:teuthology.results:Tests finished! gathering results... 2022-05-30T21:46:29.476 INFO:scrape:Found 33 jobs 2022-05-30T21:46:30.043 INFO:scrape:Found 1 distinct failure reasons 2022-05-30T21:46:30.044 INFO:scrape:Failure: "2022-05-30T20:23:10.648548+0000 mon.a (mon.0) 398 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2022-05-30T21:46:30.044 INFO:scrape:1 jobs: ['6853704'] 2022-05-30T21:46:30.044 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/snap-schedule}'] 2022-05-30T21:46:30.044 INFO:scrape: 2022-05-30T21:46:30.046 INFO:teuthology.results:Sending results to ridave@redhat.com: Test Run: rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/ failed: 1 dead: 0 running: 0 waiting: 0 queued: 0 passed: 32 Fail ================================================================= [6853704] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:53:43 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853704/ log: http://qa-proxy.ceph.com/teuthology/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853704/ "2022-05-30T20:23:10.648548+0000 mon.a (mon.0) 398 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log Pass ================================================================= [6853694] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:14:29 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853694/ [6853695] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:28:08 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853695/ [6853696] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 00:44:11 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853696/ [6853697] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:23:31 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853697/ [6853698] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:10:45 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853698/ [6853699] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} time: 00:25:50 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853699/ [6853700] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:18:58 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853700/ [6853701] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:19:09 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853701/ [6853702] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:11:00 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853702/ [6853703] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:20:54 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853703/ [6853705] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:34:49 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853705/ [6853706] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:29:16 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853706/ [6853707] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} time: 00:27:18 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853707/ [6853708] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:27 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853708/ [6853709] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:24:52 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853709/ [6853710] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:25 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853710/ [6853711] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:29:32 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853711/ [6853712] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx}} time: 00:33:02 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853712/ [6853713] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:16:32 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853713/ [6853714] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:02 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853714/ [6853715] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:19:16 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853715/ [6853716] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:52:38 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853716/ [6853717] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:34:39 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853717/ [6853718] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:20:56 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853718/ [6853719] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:32 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853719/ [6853720] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:33:13 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853720/ [6853721] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:15:39 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853721/ [6853722] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:31:28 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853722/ [6853723] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:18:03 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853723/ [6853724] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:18:23 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853724/ [6853725] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:22 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853725/ [6853726] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:12:26 info: http://pulpito.ceph.com/rishabh-2022-05-30_19:20:11-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6853726/