2023-01-10T00:16:12.323 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2023-01-10T00:16:12.324 INFO:teuthology.results:seed: '7050' 2023-01-10T03:25:02.978 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2023-01-10T03:25:03.005 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-01-10T06:24:16.339 INFO:teuthology.results:Tests finished! gathering results... 2023-01-10T06:24:19.309 INFO:scrape:Found 32 jobs 2023-01-10T06:24:26.235 INFO:scrape:Found 1 distinct failure reasons 2023-01-10T06:24:26.236 INFO:scrape:Failure: "2023-01-10T02:42:43.676232+0000 mgr.x (mgr.34209) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log 2023-01-10T06:24:26.236 INFO:scrape:25 jobs: ['7132594', '7132605', '7132603', '7132587', '7132580', '7132589', '7132583', '7132582', '7132599', '7132601', '7132581', '7132592', '7132596', '7132597', '7132576', '7132578', '7132593', '7132588', '7132598', '7132577', '7132585', '7132595', '7132584', '7132600', '7132604'] 2023-01-10T06:24:26.237 INFO:scrape:suites intersection: ['start}', 'ubuntu_18.04}'] 2023-01-10T06:24:26.238 INFO:scrape:suites union: ['1-ceph-install/nautilus', '1-nautilus-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '2-workload/{blogbench', '3-thrash/default', '3-upgrade-sequence/upgrade-all', '3-upgrade-sequence/upgrade-mon-osd-mds', '3.1-objectstore/bluestore-bitmap', '3.1-objectstore/filestore-xfs', '4-ec-workload/{rados-ec-workload', '4-pacific', '4-workload/{radosbench', '5-final-workload/{blogbench', '5-finish-upgrade', '6-pacific', '7-final-workload', '7-msgr2', '8-final-workload/{rbd-python', 'ec-rados-default', 'mon_election/classic', 'mon_election/connectivity', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd-cls', 'rbd-ec-workload}', 'rbd-import-export', 'rbd_api', 'rbd_cls', 'rbd_import_export', 'readwrite', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'test_rbd_api', 'test_rbd_python}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:nautilus-x/parallel/{0-cluster/{openstack', 'upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack', 'upgrade:nautilus-x/stress-split/{0-cluster/{openstack'] 2023-01-10T06:24:26.238 INFO:scrape: 2023-01-10T06:24:26.242 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/ failed: 25 dead: 0 running: 0 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [7132576] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 02:47:48 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132576/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132576/ "2023-01-10T00:44:36.808323+0000 mgr.x (mgr.94170) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132577] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 01:53:21 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132577/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132577/ "2023-01-10T00:46:34.898279+0000 mgr.x (mgr.54171) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132578] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/filestore-xfs ubuntu_18.04} ----------------------------------------------------------------- time: 02:11:34 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132578/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132578/ "2023-01-10T00:46:00.386651+0000 mgr.x (mgr.74220) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132580] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 02:43:40 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132580/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132580/ "2023-01-10T00:53:03.573280+0000 mgr.x (mgr.64170) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132581] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 01:16:49 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132581/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132581/ "2023-01-10T00:52:50.160306+0000 mgr.x (mgr.94158) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132582] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/filestore-xfs ubuntu_18.04} ----------------------------------------------------------------- time: 02:10:31 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132582/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132582/ "2023-01-10T00:54:11.955384+0000 mgr.x (mgr.24225) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132583] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/bluestore-bitmap ubuntu_18.04} ----------------------------------------------------------------- time: 02:16:23 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132583/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132583/ "2023-01-10T00:58:59.283103+0000 mgr.x (mgr.4753) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132584] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 02:48:47 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132584/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132584/ "2023-01-10T01:02:32.827808+0000 mgr.x (mgr.54103) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132585] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 01:37:50 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132585/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132585/ "2023-01-10T01:05:30.430316+0000 mgr.x (mgr.94098) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132587] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/bluestore-bitmap ubuntu_18.04} ----------------------------------------------------------------- time: 02:16:24 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132587/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132587/ "2023-01-10T01:08:39.643681+0000 mgr.x (mgr.24216) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132588] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 02:47:32 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132588/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132588/ "2023-01-10T01:09:25.431178+0000 mgr.x (mgr.94098) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132589] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 01:35:24 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132589/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132589/ "2023-01-10T01:25:50.076336+0000 mgr.x (mgr.94170) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132592] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 02:51:41 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132592/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132592/ "2023-01-10T02:21:21.886671+0000 mgr.x (mgr.64167) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132593] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 01:32:34 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132593/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132593/ "2023-01-10T02:36:47.509313+0000 mgr.x (mgr.64098) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132594] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/filestore-xfs ubuntu_18.04} ----------------------------------------------------------------- time: 02:09:56 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132594/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132594/ "2023-01-10T02:42:43.676232+0000 mgr.x (mgr.34209) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132595] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/bluestore-bitmap ubuntu_18.04} ----------------------------------------------------------------- time: 02:14:33 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132595/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132595/ "2023-01-10T02:49:46.930843+0000 mgr.x (mgr.34110) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132596] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 02:58:14 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132596/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132596/ "2023-01-10T02:52:51.736697+0000 mgr.x (mgr.54174) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132597] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 01:35:08 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132597/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132597/ "2023-01-10T03:06:40.046476+0000 mgr.x (mgr.54174) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132598] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/filestore-xfs ubuntu_18.04} ----------------------------------------------------------------- time: 02:10:13 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132598/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132598/ "2023-01-10T03:06:50.272194+0000 mgr.x (mgr.4808) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132599] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/bluestore-bitmap ubuntu_18.04} ----------------------------------------------------------------- time: 02:15:42 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132599/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132599/ "2023-01-10T03:10:49.179900+0000 mgr.x (mgr.44220) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132600] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 02:52:25 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132600/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132600/ "2023-01-10T03:13:57.347487+0000 mgr.x (mgr.94158) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132601] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 01:17:19 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132601/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132601/ "2023-01-10T03:24:52.268525+0000 mgr.x (mgr.54174) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132603] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/bluestore-bitmap ubuntu_18.04} ----------------------------------------------------------------- time: 02:14:05 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132603/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132603/ "2023-01-10T03:37:07.866879+0000 mgr.x (mgr.34212) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132604] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 02:53:58 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132604/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132604/ "2023-01-10T03:41:40.723106+0000 mgr.x (mgr.94179) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7132605] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 01:51:07 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132605/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132605/ "2023-01-10T03:45:14.894184+0000 mgr.x (mgr.54168) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log Pass ================================================================= [7132575] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/bluestore-bitmap ubuntu_18.04} time: 02:16:48 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132575/ [7132579] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/bluestore-bitmap ubuntu_18.04} time: 02:17:19 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132579/ [7132586] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/filestore-xfs ubuntu_18.04} time: 02:09:23 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132586/ [7132590] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/filestore-xfs ubuntu_18.04} time: 02:09:36 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132590/ [7132591] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/bluestore-bitmap ubuntu_18.04} time: 02:15:31 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132591/ [7132602] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/filestore-xfs ubuntu_18.04} time: 02:11:13 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132602/ [7132606] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/filestore-xfs ubuntu_18.04} time: 02:10:58 info: http://pulpito.ceph.com/teuthology-2023-01-07_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/7132606/