2024-02-15T02:32:02.915 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-15T02:32:02.915 INFO:teuthology.results:seed: '8767' 2024-02-15T07:25:24.715 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-15T07:25:24.731 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-15T10:57:42.032 INFO:teuthology.results:Tests finished! gathering results... 2024-02-15T10:57:47.957 INFO:scrape:Found 74 jobs 2024-02-15T10:57:50.301 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560680/teuthology.log 2024-02-15T10:58:10.291 INFO:scrape:Found 4 distinct failure reasons 2024-02-15T10:58:10.291 INFO:scrape:Failure: "2024-02-15T03:02:24.913518+0000 mon.a (mon.0) 181 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log 2024-02-15T10:58:10.291 INFO:scrape:56 jobs 2024-02-15T10:58:10.292 INFO:scrape:suites intersection: [] 2024-02-15T10:58:10.292 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:reef-x/parallel/{0-random-distro$/{centos_9.stream}', 'upgrade:reef-x/parallel/{0-random-distro$/{ubuntu_22.04}', 'upgrade:reef-x/stress-split/{0-distro/centos_9.stream', 'upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04', 'workload/{ec-rados-default'] 2024-02-15T10:58:10.292 INFO:scrape: 2024-02-15T10:58:10.292 INFO:scrape:Dead: None 2024-02-15T10:58:10.293 INFO:scrape:4 jobs: ['7560680', '7560660', '7560670', '7560667'] 2024-02-15T10:58:10.293 INFO:scrape: 2024-02-15T10:58:10.293 INFO:scrape:Failure: timeout expired in wait_until_healthy 2024-02-15T10:58:10.293 INFO:scrape:13 jobs: ['7560672', '7560638', '7560668', '7560679', '7560635', '7560624', '7560675', '7560650', '7560642', '7560661', '7560657', '7560627', '7560645'] 2024-02-15T10:58:10.293 INFO:scrape:suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04'] 2024-02-15T10:58:10.293 INFO:scrape:suites union: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04'] 2024-02-15T10:58:10.293 INFO:scrape: 2024-02-15T10:58:10.293 INFO:scrape:Failure: Command failed on smithi006 with status 127: "/bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write'" 2024-02-15T10:58:10.294 INFO:scrape:1 jobs: ['7560631'] 2024-02-15T10:58:10.294 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/rbd-import-export', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04'] 2024-02-15T10:58:10.294 INFO:scrape: 2024-02-15T10:58:10.299 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/ failed: 70 dead: 4 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7560610] upgrade:reef-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:44:12 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560610/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560610/ "2024-02-15T03:10:00.000117+0000 mon.a (mon.0) 854 : cluster 3 [WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid" in cluster log [7560611] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:13:48 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560611/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560611/ "2024-02-15T03:09:21.341313+0000 mon.a (mon.0) 119 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560612] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 02:38:46 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560612/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560612/ "2024-02-15T03:20:00.000142+0000 mon.a (mon.0) 900 : cluster 3 [WRN] OSD_DOWN: 1 osds down" in cluster log [7560613] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:11:20 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560613/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560613/ "2024-02-15T03:02:24.913518+0000 mon.a (mon.0) 181 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560614] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 03:01:31 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560614/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560614/ "2024-02-15T03:08:51.627504+0000 mon.a (mon.0) 8 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7560615] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:11:54 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560615/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560615/ "2024-02-15T03:10:35.836147+0000 mon.a (mon.0) 64 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560616] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 02:58:46 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560616/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560616/ "2024-02-15T03:03:45.196673+0000 mon.a (mon.0) 181 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560617] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:30:56 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560617/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560617/ "2024-02-15T03:08:21.744331+0000 mon.a (mon.0) 181 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560618] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 02:26:29 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560618/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560618/ "2024-02-15T03:20:00.000177+0000 mon.a (mon.0) 404 : cluster 3 [WRN] OSD_DOWN: 1 osds down" in cluster log [7560619] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:16:17 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560619/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560619/ "2024-02-15T03:20:00.000136+0000 mon.a (mon.0) 730 : cluster 4 [ERR] MDS_ALL_DOWN: 1 filesystem is offline" in cluster log [7560620] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 03:16:44 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560620/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560620/ "2024-02-15T03:48:51.448204+0000 mon.a (mon.0) 8 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7560621] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 02:59:02 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560621/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560621/ "2024-02-15T03:40:46.573403+0000 mon.a (mon.0) 160 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560622] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 02:43:24 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560622/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560622/ "2024-02-15T03:50:00.000159+0000 mon.a (mon.0) 553 : cluster 3 [WRN] OSD_DOWN: 1 osds down" in cluster log [7560623] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:22:55 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560623/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560623/ "2024-02-15T04:10:00.000150+0000 mon.a (mon.0) 1321 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log [7560624] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 00:44:57 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560624/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560624/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3a2ecc3580404ebdb9e244623a79c3e4 timeout expired in wait_until_healthy [7560625] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 02:56:05 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560625/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560625/ "2024-02-15T03:58:30.726708+0000 mon.a (mon.0) 174 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560626] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 03:05:07 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560626/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560626/ "2024-02-15T04:07:02.504704+0000 mon.a (mon.0) 8 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7560627] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 00:44:05 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560627/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560627/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c18e072e67804781818f3a92d5e53ff9 timeout expired in wait_until_healthy [7560628] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 02:46:14 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560628/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560628/ "2024-02-15T05:20:00.000132+0000 mon.a (mon.0) 1632 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 25484/211146 objects degraded (12.069%), 6 pgs degraded, 6 pgs undersized" in cluster log [7560629] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 03:13:43 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560629/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560629/ "2024-02-15T05:08:28.570121+0000 mon.a (mon.0) 126 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560630] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 02:42:39 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560630/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560630/ "2024-02-15T05:17:57.350950+0000 mon.a (mon.0) 101 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560631] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:46:44 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560631/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560631/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1edbdd63db2d46f09b6508864f1d3341 Command failed on smithi006 with status 127: "/bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to- stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write'" [7560632] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 03:06:49 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560632/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560632/ "2024-02-15T06:03:59.483210+0000 mon.a (mon.0) 8 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7560633] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 03:05:42 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560633/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560633/ "2024-02-15T06:20:00.000121+0000 mon.a (mon.0) 993 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log [7560634] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 02:56:14 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560634/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560634/ "2024-02-15T06:13:35.942467+0000 mon.a (mon.0) 182 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560635] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:56:02 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560635/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560635/ sentry: https://sentry.ceph.com/organizations/ceph/?query=528c076781cb4e3da4ae462e1aa0303b timeout expired in wait_until_healthy [7560636] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 02:45:54 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560636/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560636/ "2024-02-15T06:40:00.000143+0000 mon.a (mon.0) 1202 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log [7560637] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 03:01:37 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560637/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560637/ "2024-02-15T06:23:44.847804+0000 mon.a (mon.0) 102 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560638] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 01:11:19 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560638/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560638/ sentry: https://sentry.ceph.com/organizations/ceph/?query=53082ed2de76408ab9c4496b6d4d1a95 timeout expired in wait_until_healthy [7560639] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 02:46:53 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560639/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560639/ "2024-02-15T06:50:00.000153+0000 mon.a (mon.0) 1513 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log [7560640] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 02:41:12 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560640/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560640/ "2024-02-15T06:40:00.000218+0000 mon.a (mon.0) 1640 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log [7560641] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 03:26:33 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560641/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560641/ "2024-02-15T06:34:01.708891+0000 mon.a (mon.0) 115 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560642] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:44:56 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560642/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560642/ sentry: https://sentry.ceph.com/organizations/ceph/?query=45ebdb1b63004b8aa950394848b29ff1 timeout expired in wait_until_healthy [7560643] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 03:07:10 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560643/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560643/ "2024-02-15T06:33:13.591728+0000 mon.a (mon.0) 97 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560644] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 03:07:12 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560644/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560644/ "2024-02-15T06:32:57.755333+0000 mon.a (mon.0) 8 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7560645] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:43:39 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560645/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560645/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e43d7e2ae4ba4ba7b2f1585eef6c8b83 timeout expired in wait_until_healthy [7560646] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 03:01:16 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560646/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560646/ "2024-02-15T06:40:00.000224+0000 mon.a (mon.0) 773 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 19/342 objects degraded (5.556%), 6 pgs degraded" in cluster log [7560647] upgrade:reef-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 02:03:01 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560647/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560647/ "2024-02-15T06:29:07.324682+0000 mon.a (mon.0) 181 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560648] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 03:01:49 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560648/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560648/ "2024-02-15T07:00:00.000135+0000 mon.a (mon.0) 1346 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 10873/83232 objects degraded (13.063%), 3 pgs degraded, 3 pgs undersized" in cluster log [7560649] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 02:58:08 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560649/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560649/ "2024-02-15T06:36:21.921026+0000 mon.a (mon.0) 100 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560650] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:44:18 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560650/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560650/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2a46518d03b84c838a7edc0e851f1368 timeout expired in wait_until_healthy [7560651] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 03:16:53 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560651/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560651/ "2024-02-15T06:37:39.374665+0000 mon.a (mon.0) 8 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7560652] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 02:37:36 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560652/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560652/ "2024-02-15T06:27:22.210150+0000 mon.a (mon.0) 173 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560653] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 03:24:34 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560653/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560653/ "2024-02-15T06:56:56.476150+0000 mon.a (mon.0) 206 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560654] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 03:17:41 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560654/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560654/ "2024-02-15T06:38:48.838660+0000 mon.a (mon.0) 184 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560655] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 02:59:16 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560655/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560655/ "2024-02-15T06:47:59.484451+0000 mon.a (mon.0) 144 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560656] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 02:26:57 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560656/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560656/ "2024-02-15T06:52:27.877108+0000 mon.a (mon.0) 102 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560657] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 01:12:41 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560657/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560657/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ec11b20aeb2446b891eded30c9dd6d5a timeout expired in wait_until_healthy [7560658] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 02:22:22 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560658/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560658/ "2024-02-15T07:00:00.002263+0000 mon.a (mon.0) 547 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log [7560659] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 03:17:54 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560659/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560659/ "2024-02-15T06:52:07.784276+0000 mon.a (mon.0) 267 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560661] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:45:04 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560661/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560661/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3767f0708c56415dbb3523edc48602c4 timeout expired in wait_until_healthy [7560662] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 02:38:09 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560662/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560662/ "2024-02-15T07:20:00.000088+0000 mon.a (mon.0) 849 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log [7560663] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 03:24:58 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560663/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560663/ "2024-02-15T07:06:04.564014+0000 mon.a (mon.0) 8 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7560664] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 02:52:10 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560664/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560664/ "2024-02-15T06:59:52.635313+0000 mon.a (mon.0) 184 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560665] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 03:03:03 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560665/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560665/ "2024-02-15T07:06:41.986406+0000 mon.a (mon.0) 194 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560666] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 03:15:00 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560666/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560666/ "2024-02-15T07:08:48.837493+0000 mon.a (mon.0) 71 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560668] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 00:44:11 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560668/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560668/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ac951b2403a9414ea82e1af102ea0de7 timeout expired in wait_until_healthy [7560669] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:32:57 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560669/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560669/ "2024-02-15T07:27:19.598294+0000 mon.a (mon.0) 8 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7560671] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:35:44 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560671/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560671/ "2024-02-15T07:28:45.497801+0000 mon.a (mon.0) 181 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560672] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 00:54:47 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560672/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560672/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3a4a25b4ab054c3593c6270764c39b24 timeout expired in wait_until_healthy [7560673] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:05:55 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560673/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560673/ "2024-02-15T07:39:53.426450+0000 mon.a (mon.0) 162 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7560674] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 02:35:38 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560674/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560674/ "2024-02-15T07:50:00.000146+0000 mon.a (mon.0) 670 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log [7560675] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 01:11:43 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560675/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560675/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b2a6811daa844abe8120c896bbbca5c9 timeout expired in wait_until_healthy [7560676] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 02:28:12 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560676/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560676/ "2024-02-15T07:50:00.000142+0000 mon.a (mon.0) 597 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log [7560677] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:15:49 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560677/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560677/ "2024-02-15T08:00:00.001147+0000 mon.a (mon.0) 1807 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 8764/82218 objects degraded (10.659%), 5 pgs degraded" in cluster log [7560678] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 03:09:55 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560678/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560678/ "2024-02-15T07:40:00.264544+0000 mon.a (mon.0) 113 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560679] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 00:45:29 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560679/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560679/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa10c95e02df4e139c06ad4b12976e1e timeout expired in wait_until_healthy [7560681] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:26:11 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560681/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560681/ "2024-02-15T07:48:40.701782+0000 mon.a (mon.0) 8 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7560682] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 02:58:07 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560682/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560682/ "2024-02-15T07:44:39.208689+0000 mon.a (mon.0) 181 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7560683] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:03:12 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560683/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560683/ "2024-02-15T07:53:51.168061+0000 mon.a (mon.0) 313 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log Dead ================================================================= [7560660] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560660/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560660/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7560667] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560667/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560667/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7560670] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560670/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560670/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7560680] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560680/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-15_02:25:02-upgrade:reef-x-main-distro-default-smithi/7560680/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds