2023-10-09T15:08:50.569 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-09T15:08:50.569 INFO:teuthology.results:subset: '175/1024' 2023-10-09T15:08:50.569 INFO:teuthology.results:seed: '3539' 2023-10-09T15:09:35.507 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-09T15:09:35.510 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-09T16:54:43.055 INFO:teuthology.results:Tests finished! gathering results... 2023-10-09T16:54:43.315 INFO:scrape:Found 6 jobs 2023-10-09T16:54:46.281 INFO:scrape:Found 2 distinct failure reasons 2023-10-09T16:54:46.281 INFO:scrape:Failure: Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-10-09T16:54:46.282 INFO:scrape:1 jobs: ['7418460'] 2023-10-09T16:54:46.282 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/clone}}'] 2023-10-09T16:54:46.282 INFO:scrape: 2023-10-09T16:54:46.282 INFO:scrape:Failure: "2023-10-09T15:28:19.106458+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log 2023-10-09T16:54:46.282 INFO:scrape:2 jobs: ['7418462', '7418459'] 2023-10-09T16:54:46.283 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/volumes/{overrides'] 2023-10-09T16:54:46.283 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/volumes/{overrides', 'test/basic}}', 'test/snapshot}}'] 2023-10-09T16:54:46.283 INFO:scrape: 2023-10-09T16:54:46.284 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-10-09_15:08:42-fs-quincy-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-10-09_15:08:42-fs-quincy-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-09_15:08:42-fs-quincy-testing-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 3 Fail ================================================================= [7418459] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 01:24:56 info: http://pulpito.ceph.com/vshankar-2023-10-09_15:08:42-fs-quincy-testing-default-smithi/7418459/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-09_15:08:42-fs-quincy-testing-default-smithi/7418459/ "2023-10-09T15:29:42.425526+0000 mgr.x (mgr.4102) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7418460] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:59:34 info: http://pulpito.ceph.com/vshankar-2023-10-09_15:08:42-fs-quincy-testing-default-smithi/7418460/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-09_15:08:42-fs-quincy-testing-default-smithi/7418460/ sentry: https://sentry.ceph.com/organizations/ceph/?query=253d55057cfa4c0f8e716ad3abfe37cf Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) [7418462] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} ----------------------------------------------------------------- time: 00:31:05 info: http://pulpito.ceph.com/vshankar-2023-10-09_15:08:42-fs-quincy-testing-default-smithi/7418462/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-09_15:08:42-fs-quincy-testing-default-smithi/7418462/ "2023-10-09T15:28:19.106458+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log Pass ================================================================= [7418458] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:35:07 info: http://pulpito.ceph.com/vshankar-2023-10-09_15:08:42-fs-quincy-testing-default-smithi/7418458/ [7418461] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:23:02 info: http://pulpito.ceph.com/vshankar-2023-10-09_15:08:42-fs-quincy-testing-default-smithi/7418461/ [7418463] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:32:32 info: http://pulpito.ceph.com/vshankar-2023-10-09_15:08:42-fs-quincy-testing-default-smithi/7418463/