2024-03-22T09:57:10.793 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-22T09:57:10.793 INFO:teuthology.results:seed: '3902' 2024-03-22T09:57:15.624 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-22T09:57:15.624 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-22T12:25:26.633 INFO:teuthology.results:Tests finished! gathering results... 2024-03-22T12:25:27.060 INFO:scrape:Found 12 jobs 2024-03-22T12:25:30.450 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616247/teuthology.log 2024-03-22T12:25:32.787 INFO:scrape:Found 2 distinct failure reasons 2024-03-22T12:25:32.788 INFO:scrape:Failure: "2024-03-22T10:15:08.749104+0000 mon.a (mon.0) 508 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2024-03-22T12:25:32.788 INFO:scrape:11 jobs: ['7616249', '7616252', '7616255', '7616251', '7616258', '7616254', '7616248', '7616250', '7616256', '7616257', '7616253'] 2024-03-22T12:25:32.788 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/snap-schedule}'] 2024-03-22T12:25:32.788 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'subvol_versions/create_subvol_version_v2', 'tasks/snap-schedule}'] 2024-03-22T12:25:32.788 INFO:scrape: 2024-03-22T12:25:32.788 INFO:scrape:Dead: None 2024-03-22T12:25:32.788 INFO:scrape:1 jobs: ['7616247'] 2024-03-22T12:25:32.789 INFO:scrape: 2024-03-22T12:25:32.789 INFO:teuthology.results:Sending results to mchangir@redhat.com: Test Run: mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/ failed: 11 dead: 1 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7616248] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} ----------------------------------------------------------------- time: 02:13:29 info: https://pulpito.ceph.com/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616248/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616248/ "2024-03-22T10:24:58.738181+0000 mon.a (mon.0) 398 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY)" in cluster log [7616249] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} ----------------------------------------------------------------- time: 02:05:16 info: https://pulpito.ceph.com/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616249/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616249/ "2024-03-22T10:15:08.749104+0000 mon.a (mon.0) 508 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7616250] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap-schedule} ----------------------------------------------------------------- time: 02:15:04 info: https://pulpito.ceph.com/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616250/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616250/ "2024-03-22T10:35:23.146058+0000 mon.a (mon.0) 569 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7616251] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap-schedule} ----------------------------------------------------------------- time: 02:12:24 info: https://pulpito.ceph.com/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616251/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616251/ "2024-03-22T10:23:07.143463+0000 mon.a (mon.0) 511 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7616252] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap-schedule} ----------------------------------------------------------------- time: 02:04:55 info: https://pulpito.ceph.com/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616252/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616252/ "2024-03-22T10:17:40.036071+0000 mon.a (mon.0) 395 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log [7616253] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} ----------------------------------------------------------------- time: 02:13:52 info: https://pulpito.ceph.com/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616253/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616253/ "2024-03-22T10:34:30.544539+0000 mon.a (mon.0) 661 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log [7616254] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} ----------------------------------------------------------------- time: 02:11:19 info: https://pulpito.ceph.com/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616254/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616254/ "2024-03-22T10:35:22.686561+0000 mon.a (mon.0) 494 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log [7616255] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} ----------------------------------------------------------------- time: 02:04:13 info: https://pulpito.ceph.com/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616255/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616255/ "2024-03-22T10:29:15.905101+0000 mon.a (mon.0) 668 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log [7616256] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap-schedule} ----------------------------------------------------------------- time: 02:14:22 info: https://pulpito.ceph.com/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616256/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616256/ "2024-03-22T10:33:30.506745+0000 mon.a (mon.0) 681 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log [7616257] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap-schedule} ----------------------------------------------------------------- time: 02:12:46 info: https://pulpito.ceph.com/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616257/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616257/ "2024-03-22T10:34:28.263802+0000 mon.a (mon.0) 673 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log [7616258] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap-schedule} ----------------------------------------------------------------- time: 02:06:13 info: https://pulpito.ceph.com/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616258/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616258/ "2024-03-22T10:32:12.494837+0000 mon.a (mon.0) 647 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log Dead ================================================================= [7616247] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616247/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-22_09:41:51-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616247/ Error reimaging machines: Failed to power on smithi018