2023-03-22T13:14:53.828 INFO:root:teuthology version: 0.0.1.dev66+g73b89aa
2023-03-22T13:14:53.829 INFO:teuthology.results:seed: '586'
2023-03-22T13:33:07.272 INFO:root:teuthology version: 0.0.1.dev66+g73b89aa
2023-03-22T13:33:07.275 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-03-22T16:55:23.231 INFO:teuthology.results:Tests finished! gathering results...
2023-03-22T16:55:24.590 INFO:scrape:Found 15 jobs
2023-03-22T16:55:24.711 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-3075-g9cd18b46 (9cd18b464c3f51a46a43f96ca21813a9b81c2781) reef (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f06ca93f7ff]
,  2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06ca93fa2a]

2023-03-22T16:55:24.715 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 0 to deep-scrub
, 2023-03-22T14:04:52.586 INFO:tasks.ceph:Scrubbing osd.1
, 2023-03-22T14:04:52.586 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0

2023-03-22T16:55:24.716 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:04:54.103 INFO:tasks.ceph:pgid 16.5 last_scrub_stamp 2023-03-22T13:52:27.477405+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=52, tm_sec=27, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.103 INFO:tasks.ceph:pgid 15.1a last_scrub_stamp 2023-03-22T13:52:25.461801+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=52, tm_sec=25, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.104 INFO:tasks.ceph:pgid 14.1b last_scrub_stamp 2023-03-22T13:49:59.539533+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=49, tm_sec=59, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)

2023-03-22T16:55:24.716 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:04:54.133 INFO:tasks.ceph:pgid 17.2 last_scrub_stamp 2023-03-22T13:59:10.343241+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=59, tm_sec=10, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.133 INFO:tasks.ceph:pgid 16.3 last_scrub_stamp 2023-03-22T13:52:27.477405+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=52, tm_sec=27, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.133 INFO:tasks.ceph:pgid 15.1c last_scrub_stamp 2023-03-22T13:52:25.461801+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=52, tm_sec=25, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)

2023-03-22T16:55:24.717 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:04:54.164 INFO:tasks.ceph:pgid 18.1b last_scrub_stamp 2023-03-22T13:59:26.568516+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=59, tm_sec=26, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.164 INFO:tasks.ceph:pgid 17.18 last_scrub_stamp 2023-03-22T13:59:10.343241+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=59, tm_sec=10, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.164 INFO:tasks.ceph:pgid 16.19 last_scrub_stamp 2023-03-22T13:52:27.477405+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=52, tm_sec=27, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)

2023-03-22T16:55:24.717 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:04:54.194 INFO:tasks.ceph:pgid 11.8 last_scrub_stamp 2023-03-22T13:41:48.532315+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=41, tm_sec=48, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.194 INFO:tasks.ceph:pgid 12.f last_scrub_stamp 2023-03-22T13:42:12.535929+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=42, tm_sec=12, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.194 INFO:tasks.ceph:pgid 20.15 last_scrub_stamp 2023-03-22T14:01:17.682111+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=1, tm_sec=17, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)

2023-03-22T16:55:24.721 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:04:54.223 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2023-03-22T13:39:52.813992+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=39, tm_sec=52, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.224 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2023-03-22T13:39:46.715017+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=39, tm_sec=46, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.224 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2023-03-22T13:39:40.514091+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=39, tm_sec=40, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)

2023-03-22T16:55:24.721 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:05:27.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored
, 2023-03-22T14:05:32.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s
, 2023-03-22T14:05:33.350 INFO:tasks.ceph.osd.2:Stopped

2023-03-22T16:55:24.722 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:06:43.214 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2023-03-22T14:06:43.214 ERROR:teuthology.run_tasks:Manager failed: internal.archive
, Traceback (most recent call last):

2023-03-22T16:55:24.748 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-3075-g9cd18b46 (9cd18b464c3f51a46a43f96ca21813a9b81c2781) reef (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f06ca93f7ff]
,  2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06ca93fa2a]

2023-03-22T16:55:24.752 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 0 to deep-scrub
, 2023-03-22T14:04:52.586 INFO:tasks.ceph:Scrubbing osd.1
, 2023-03-22T14:04:52.586 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0

2023-03-22T16:55:24.752 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:04:54.103 INFO:tasks.ceph:pgid 16.5 last_scrub_stamp 2023-03-22T13:52:27.477405+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=52, tm_sec=27, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.103 INFO:tasks.ceph:pgid 15.1a last_scrub_stamp 2023-03-22T13:52:25.461801+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=52, tm_sec=25, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.104 INFO:tasks.ceph:pgid 14.1b last_scrub_stamp 2023-03-22T13:49:59.539533+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=49, tm_sec=59, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)

2023-03-22T16:55:24.753 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:04:54.133 INFO:tasks.ceph:pgid 17.2 last_scrub_stamp 2023-03-22T13:59:10.343241+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=59, tm_sec=10, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.133 INFO:tasks.ceph:pgid 16.3 last_scrub_stamp 2023-03-22T13:52:27.477405+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=52, tm_sec=27, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.133 INFO:tasks.ceph:pgid 15.1c last_scrub_stamp 2023-03-22T13:52:25.461801+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=52, tm_sec=25, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)

2023-03-22T16:55:24.753 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:04:54.164 INFO:tasks.ceph:pgid 18.1b last_scrub_stamp 2023-03-22T13:59:26.568516+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=59, tm_sec=26, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.164 INFO:tasks.ceph:pgid 17.18 last_scrub_stamp 2023-03-22T13:59:10.343241+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=59, tm_sec=10, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.164 INFO:tasks.ceph:pgid 16.19 last_scrub_stamp 2023-03-22T13:52:27.477405+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=52, tm_sec=27, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)

2023-03-22T16:55:24.754 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:04:54.194 INFO:tasks.ceph:pgid 11.8 last_scrub_stamp 2023-03-22T13:41:48.532315+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=41, tm_sec=48, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.194 INFO:tasks.ceph:pgid 12.f last_scrub_stamp 2023-03-22T13:42:12.535929+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=42, tm_sec=12, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.194 INFO:tasks.ceph:pgid 20.15 last_scrub_stamp 2023-03-22T14:01:17.682111+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=1, tm_sec=17, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)

2023-03-22T16:55:24.757 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:04:54.223 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2023-03-22T13:39:52.813992+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=39, tm_sec=52, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.224 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2023-03-22T13:39:46.715017+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=39, tm_sec=46, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.224 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2023-03-22T13:39:40.514091+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=39, tm_sec=40, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)

2023-03-22T16:55:24.757 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:05:27.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored
, 2023-03-22T14:05:32.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s
, 2023-03-22T14:05:33.350 INFO:tasks.ceph.osd.2:Stopped

2023-03-22T16:55:24.758 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:06:43.214 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2023-03-22T14:06:43.214 ERROR:teuthology.run_tasks:Manager failed: internal.archive
, Traceback (most recent call last):

2023-03-22T16:55:24.785 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-3075-g9cd18b46 (9cd18b464c3f51a46a43f96ca21813a9b81c2781) reef (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f06ca93f7ff]
,  2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06ca93fa2a]

2023-03-22T16:55:24.788 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 0 to deep-scrub
, 2023-03-22T14:04:52.586 INFO:tasks.ceph:Scrubbing osd.1
, 2023-03-22T14:04:52.586 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0

2023-03-22T16:55:24.789 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:04:54.103 INFO:tasks.ceph:pgid 16.5 last_scrub_stamp 2023-03-22T13:52:27.477405+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=52, tm_sec=27, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.103 INFO:tasks.ceph:pgid 15.1a last_scrub_stamp 2023-03-22T13:52:25.461801+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=52, tm_sec=25, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.104 INFO:tasks.ceph:pgid 14.1b last_scrub_stamp 2023-03-22T13:49:59.539533+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=49, tm_sec=59, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)

2023-03-22T16:55:24.789 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:04:54.133 INFO:tasks.ceph:pgid 17.2 last_scrub_stamp 2023-03-22T13:59:10.343241+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=59, tm_sec=10, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.133 INFO:tasks.ceph:pgid 16.3 last_scrub_stamp 2023-03-22T13:52:27.477405+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=52, tm_sec=27, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.133 INFO:tasks.ceph:pgid 15.1c last_scrub_stamp 2023-03-22T13:52:25.461801+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=52, tm_sec=25, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)

2023-03-22T16:55:24.790 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:04:54.164 INFO:tasks.ceph:pgid 18.1b last_scrub_stamp 2023-03-22T13:59:26.568516+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=59, tm_sec=26, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.164 INFO:tasks.ceph:pgid 17.18 last_scrub_stamp 2023-03-22T13:59:10.343241+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=59, tm_sec=10, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.164 INFO:tasks.ceph:pgid 16.19 last_scrub_stamp 2023-03-22T13:52:27.477405+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=52, tm_sec=27, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)

2023-03-22T16:55:24.790 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:04:54.194 INFO:tasks.ceph:pgid 11.8 last_scrub_stamp 2023-03-22T13:41:48.532315+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=41, tm_sec=48, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.194 INFO:tasks.ceph:pgid 12.f last_scrub_stamp 2023-03-22T13:42:12.535929+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=42, tm_sec=12, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.194 INFO:tasks.ceph:pgid 20.15 last_scrub_stamp 2023-03-22T14:01:17.682111+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=1, tm_sec=17, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)

2023-03-22T16:55:24.793 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:04:54.223 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2023-03-22T13:39:52.813992+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=39, tm_sec=52, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.224 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2023-03-22T13:39:46.715017+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=39, tm_sec=46, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)
, 2023-03-22T14:04:54.224 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2023-03-22T13:39:40.514091+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=39, tm_sec=40, tm_wday=2, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=22, tm_hour=14, tm_min=4, tm_sec=50, tm_wday=2, tm_yday=81, tm_isdst=0)

2023-03-22T16:55:24.793 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:05:27.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored
, 2023-03-22T14:05:32.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s
, 2023-03-22T14:05:33.350 INFO:tasks.ceph.osd.2:Stopped

2023-03-22T16:55:24.794 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-22T14:06:43.214 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2023-03-22T14:06:43.214 ERROR:teuthology.run_tasks:Manager failed: internal.archive
, Traceback (most recent call last):

2023-03-22T16:55:24.934 INFO:scrape:Found 3 distinct failure reasons
2023-03-22T16:55:24.935 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_lock_fence.sh
2023-03-22T16:55:24.936 INFO:scrape:1 jobs: ['7216284']
2023-03-22T16:55:24.936 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_lock_and_fence}']
2023-03-22T16:55:24.936 INFO:scrape:
2023-03-22T16:55:24.936 INFO:scrape:Failure: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi146 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9cd18b464c3f51a46a43f96ca21813a9b81c2781 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh'
2023-03-22T16:55:24.937 INFO:scrape:1 jobs: ['7216275']
2023-03-22T16:55:24.937 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests}']
2023-03-22T16:55:24.937 INFO:scrape:
2023-03-22T16:55:24.938 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh
2023-03-22T16:55:24.938 INFO:scrape:1 jobs: ['7216287']
2023-03-22T16:55:24.938 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}']
2023-03-22T16:55:24.938 INFO:scrape: