Found 19 jobs Ignoring malparsed backtrace: ceph version 18.0.0-3708-g12877b4e (12877b4e4694861aa1f9ae444a0b4532908f015a) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f52ea4ea7ff] , 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f52ea4eaa2a] Ignoring malparsed backtrace: instructed osd(s) 0 to deep-scrub , 2023-05-03T13:54:42.663 INFO:tasks.ceph:Scrubbing osd.1 , 2023-05-03T13:54:42.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 Ignoring malparsed backtrace: 2023-05-03T13:54:44.230 INFO:tasks.ceph:pgid 16.5 last_scrub_stamp 2023-05-03T13:40:58.575963+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=58, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.230 INFO:tasks.ceph:pgid 15.1a last_scrub_stamp 2023-05-03T13:40:56.567348+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=56, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.230 INFO:tasks.ceph:pgid 14.1b last_scrub_stamp 2023-05-03T13:38:25.689937+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=38, tm_sec=25, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.259 INFO:tasks.ceph:pgid 17.2 last_scrub_stamp 2023-05-03T13:48:25.680756+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=48, tm_sec=25, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.259 INFO:tasks.ceph:pgid 16.3 last_scrub_stamp 2023-05-03T13:40:58.575963+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=58, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.260 INFO:tasks.ceph:pgid 15.1c last_scrub_stamp 2023-05-03T13:40:56.567348+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=56, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.288 INFO:tasks.ceph:pgid 18.1b last_scrub_stamp 2023-05-03T13:48:56.733267+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=48, tm_sec=56, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.289 INFO:tasks.ceph:pgid 17.18 last_scrub_stamp 2023-05-03T13:48:25.680756+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=48, tm_sec=25, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.289 INFO:tasks.ceph:pgid 16.19 last_scrub_stamp 2023-05-03T13:40:58.575963+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=58, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.318 INFO:tasks.ceph:pgid 11.8 last_scrub_stamp 2023-05-03T13:29:41.434229+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=29, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.318 INFO:tasks.ceph:pgid 12.f last_scrub_stamp 2023-05-03T13:30:13.428636+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=30, tm_sec=13, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.319 INFO:tasks.ceph:pgid 20.15 last_scrub_stamp 2023-05-03T13:50:41.726504+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=50, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.348 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2023-05-03T13:27:35.487455+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=27, tm_sec=35, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.349 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2023-05-03T13:27:29.466285+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=27, tm_sec=29, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.349 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2023-05-03T13:27:23.446544+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=27, tm_sec=23, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:55:20.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored , 2023-05-03T13:55:23.484 INFO:tasks.ceph.osd.2:Stopped , 2023-05-03T13:55:23.485 INFO:teuthology.misc:Shutting down mgr daemons... Ignoring malparsed backtrace: 2023-05-03T13:56:27.804 ERROR:teuthology.run_tasks:Manager failed: internal.archive , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/run_tasks.py", line 192, in run_tasks Ignoring malparsed backtrace: ceph version 18.0.0-3708-g12877b4e (12877b4e4694861aa1f9ae444a0b4532908f015a) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f52ea4ea7ff] , 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f52ea4eaa2a] Ignoring malparsed backtrace: instructed osd(s) 0 to deep-scrub , 2023-05-03T13:54:42.663 INFO:tasks.ceph:Scrubbing osd.1 , 2023-05-03T13:54:42.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 Ignoring malparsed backtrace: 2023-05-03T13:54:44.230 INFO:tasks.ceph:pgid 16.5 last_scrub_stamp 2023-05-03T13:40:58.575963+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=58, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.230 INFO:tasks.ceph:pgid 15.1a last_scrub_stamp 2023-05-03T13:40:56.567348+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=56, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.230 INFO:tasks.ceph:pgid 14.1b last_scrub_stamp 2023-05-03T13:38:25.689937+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=38, tm_sec=25, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.259 INFO:tasks.ceph:pgid 17.2 last_scrub_stamp 2023-05-03T13:48:25.680756+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=48, tm_sec=25, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.259 INFO:tasks.ceph:pgid 16.3 last_scrub_stamp 2023-05-03T13:40:58.575963+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=58, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.260 INFO:tasks.ceph:pgid 15.1c last_scrub_stamp 2023-05-03T13:40:56.567348+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=56, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.288 INFO:tasks.ceph:pgid 18.1b last_scrub_stamp 2023-05-03T13:48:56.733267+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=48, tm_sec=56, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.289 INFO:tasks.ceph:pgid 17.18 last_scrub_stamp 2023-05-03T13:48:25.680756+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=48, tm_sec=25, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.289 INFO:tasks.ceph:pgid 16.19 last_scrub_stamp 2023-05-03T13:40:58.575963+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=58, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.318 INFO:tasks.ceph:pgid 11.8 last_scrub_stamp 2023-05-03T13:29:41.434229+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=29, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.318 INFO:tasks.ceph:pgid 12.f last_scrub_stamp 2023-05-03T13:30:13.428636+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=30, tm_sec=13, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.319 INFO:tasks.ceph:pgid 20.15 last_scrub_stamp 2023-05-03T13:50:41.726504+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=50, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.348 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2023-05-03T13:27:35.487455+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=27, tm_sec=35, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.349 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2023-05-03T13:27:29.466285+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=27, tm_sec=29, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.349 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2023-05-03T13:27:23.446544+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=27, tm_sec=23, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:55:20.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored , 2023-05-03T13:55:23.484 INFO:tasks.ceph.osd.2:Stopped , 2023-05-03T13:55:23.485 INFO:teuthology.misc:Shutting down mgr daemons... Ignoring malparsed backtrace: 2023-05-03T13:56:27.804 ERROR:teuthology.run_tasks:Manager failed: internal.archive , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/run_tasks.py", line 192, in run_tasks Ignoring malparsed backtrace: ceph version 18.0.0-3708-g12877b4e (12877b4e4694861aa1f9ae444a0b4532908f015a) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f52ea4ea7ff] , 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f52ea4eaa2a] Ignoring malparsed backtrace: instructed osd(s) 0 to deep-scrub , 2023-05-03T13:54:42.663 INFO:tasks.ceph:Scrubbing osd.1 , 2023-05-03T13:54:42.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 Ignoring malparsed backtrace: 2023-05-03T13:54:44.230 INFO:tasks.ceph:pgid 16.5 last_scrub_stamp 2023-05-03T13:40:58.575963+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=58, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.230 INFO:tasks.ceph:pgid 15.1a last_scrub_stamp 2023-05-03T13:40:56.567348+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=56, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.230 INFO:tasks.ceph:pgid 14.1b last_scrub_stamp 2023-05-03T13:38:25.689937+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=38, tm_sec=25, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.259 INFO:tasks.ceph:pgid 17.2 last_scrub_stamp 2023-05-03T13:48:25.680756+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=48, tm_sec=25, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.259 INFO:tasks.ceph:pgid 16.3 last_scrub_stamp 2023-05-03T13:40:58.575963+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=58, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.260 INFO:tasks.ceph:pgid 15.1c last_scrub_stamp 2023-05-03T13:40:56.567348+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=56, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.288 INFO:tasks.ceph:pgid 18.1b last_scrub_stamp 2023-05-03T13:48:56.733267+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=48, tm_sec=56, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.289 INFO:tasks.ceph:pgid 17.18 last_scrub_stamp 2023-05-03T13:48:25.680756+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=48, tm_sec=25, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.289 INFO:tasks.ceph:pgid 16.19 last_scrub_stamp 2023-05-03T13:40:58.575963+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=58, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.318 INFO:tasks.ceph:pgid 11.8 last_scrub_stamp 2023-05-03T13:29:41.434229+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=29, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.318 INFO:tasks.ceph:pgid 12.f last_scrub_stamp 2023-05-03T13:30:13.428636+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=30, tm_sec=13, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.319 INFO:tasks.ceph:pgid 20.15 last_scrub_stamp 2023-05-03T13:50:41.726504+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=50, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.348 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2023-05-03T13:27:35.487455+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=27, tm_sec=35, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.349 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2023-05-03T13:27:29.466285+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=27, tm_sec=29, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.349 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2023-05-03T13:27:23.446544+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=27, tm_sec=23, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:55:20.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored , 2023-05-03T13:55:23.484 INFO:tasks.ceph.osd.2:Stopped , 2023-05-03T13:55:23.485 INFO:teuthology.misc:Shutting down mgr daemons... Ignoring malparsed backtrace: 2023-05-03T13:56:27.804 ERROR:teuthology.run_tasks:Manager failed: internal.archive , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/run_tasks.py", line 192, in run_tasks Ignoring malparsed backtrace: ceph version 18.0.0-3708-g12877b4e (12877b4e4694861aa1f9ae444a0b4532908f015a) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f52ea4ea7ff] , 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f52ea4eaa2a] Ignoring malparsed backtrace: instructed osd(s) 0 to deep-scrub , 2023-05-03T13:54:42.663 INFO:tasks.ceph:Scrubbing osd.1 , 2023-05-03T13:54:42.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 Ignoring malparsed backtrace: 2023-05-03T13:54:44.230 INFO:tasks.ceph:pgid 16.5 last_scrub_stamp 2023-05-03T13:40:58.575963+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=58, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.230 INFO:tasks.ceph:pgid 15.1a last_scrub_stamp 2023-05-03T13:40:56.567348+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=56, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.230 INFO:tasks.ceph:pgid 14.1b last_scrub_stamp 2023-05-03T13:38:25.689937+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=38, tm_sec=25, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.259 INFO:tasks.ceph:pgid 17.2 last_scrub_stamp 2023-05-03T13:48:25.680756+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=48, tm_sec=25, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.259 INFO:tasks.ceph:pgid 16.3 last_scrub_stamp 2023-05-03T13:40:58.575963+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=58, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.260 INFO:tasks.ceph:pgid 15.1c last_scrub_stamp 2023-05-03T13:40:56.567348+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=56, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.288 INFO:tasks.ceph:pgid 18.1b last_scrub_stamp 2023-05-03T13:48:56.733267+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=48, tm_sec=56, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.289 INFO:tasks.ceph:pgid 17.18 last_scrub_stamp 2023-05-03T13:48:25.680756+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=48, tm_sec=25, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.289 INFO:tasks.ceph:pgid 16.19 last_scrub_stamp 2023-05-03T13:40:58.575963+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=58, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.318 INFO:tasks.ceph:pgid 11.8 last_scrub_stamp 2023-05-03T13:29:41.434229+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=29, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.318 INFO:tasks.ceph:pgid 12.f last_scrub_stamp 2023-05-03T13:30:13.428636+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=30, tm_sec=13, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.319 INFO:tasks.ceph:pgid 20.15 last_scrub_stamp 2023-05-03T13:50:41.726504+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=50, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.348 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2023-05-03T13:27:35.487455+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=27, tm_sec=35, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.349 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2023-05-03T13:27:29.466285+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=27, tm_sec=29, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.349 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2023-05-03T13:27:23.446544+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=27, tm_sec=23, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:55:20.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored , 2023-05-03T13:55:23.484 INFO:tasks.ceph.osd.2:Stopped , 2023-05-03T13:55:23.485 INFO:teuthology.misc:Shutting down mgr daemons... Ignoring malparsed backtrace: 2023-05-03T13:56:27.804 ERROR:teuthology.run_tasks:Manager failed: internal.archive , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/run_tasks.py", line 192, in run_tasks Ignoring malparsed backtrace: ceph version 18.0.0-3708-g12877b4e (12877b4e4694861aa1f9ae444a0b4532908f015a) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f52ea4ea7ff] , 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f52ea4eaa2a] Ignoring malparsed backtrace: instructed osd(s) 0 to deep-scrub , 2023-05-03T13:54:42.663 INFO:tasks.ceph:Scrubbing osd.1 , 2023-05-03T13:54:42.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 Ignoring malparsed backtrace: 2023-05-03T13:54:44.230 INFO:tasks.ceph:pgid 16.5 last_scrub_stamp 2023-05-03T13:40:58.575963+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=58, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.230 INFO:tasks.ceph:pgid 15.1a last_scrub_stamp 2023-05-03T13:40:56.567348+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=56, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.230 INFO:tasks.ceph:pgid 14.1b last_scrub_stamp 2023-05-03T13:38:25.689937+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=38, tm_sec=25, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.259 INFO:tasks.ceph:pgid 17.2 last_scrub_stamp 2023-05-03T13:48:25.680756+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=48, tm_sec=25, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.259 INFO:tasks.ceph:pgid 16.3 last_scrub_stamp 2023-05-03T13:40:58.575963+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=58, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.260 INFO:tasks.ceph:pgid 15.1c last_scrub_stamp 2023-05-03T13:40:56.567348+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=56, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.288 INFO:tasks.ceph:pgid 18.1b last_scrub_stamp 2023-05-03T13:48:56.733267+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=48, tm_sec=56, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.289 INFO:tasks.ceph:pgid 17.18 last_scrub_stamp 2023-05-03T13:48:25.680756+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=48, tm_sec=25, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.289 INFO:tasks.ceph:pgid 16.19 last_scrub_stamp 2023-05-03T13:40:58.575963+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=40, tm_sec=58, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.318 INFO:tasks.ceph:pgid 11.8 last_scrub_stamp 2023-05-03T13:29:41.434229+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=29, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.318 INFO:tasks.ceph:pgid 12.f last_scrub_stamp 2023-05-03T13:30:13.428636+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=30, tm_sec=13, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.319 INFO:tasks.ceph:pgid 20.15 last_scrub_stamp 2023-05-03T13:50:41.726504+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=50, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:54:44.348 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2023-05-03T13:27:35.487455+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=27, tm_sec=35, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.349 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2023-05-03T13:27:29.466285+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=27, tm_sec=29, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) , 2023-05-03T13:54:44.349 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2023-05-03T13:27:23.446544+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=27, tm_sec=23, tm_wday=2, tm_yday=123, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=3, tm_hour=13, tm_min=54, tm_sec=41, tm_wday=2, tm_yday=123, tm_isdst=0) Ignoring malparsed backtrace: 2023-05-03T13:55:20.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored , 2023-05-03T13:55:23.484 INFO:tasks.ceph.osd.2:Stopped , 2023-05-03T13:55:23.485 INFO:teuthology.misc:Shutting down mgr daemons... Ignoring malparsed backtrace: 2023-05-03T13:56:27.804 ERROR:teuthology.run_tasks:Manager failed: internal.archive , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/run_tasks.py", line 192, in run_tasks Found 4 distinct failure reasons Failure: Command failed (workunit test rbd/test_librbd_python.sh) on smithi077 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=12877b4e4694861aa1f9ae444a0b4532908f015a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' 1 jobs: ['7261227'] suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}'] Failure: Command failed (workunit test rados/test.sh) on smithi153 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=12877b4e4694861aa1f9ae444a0b4532908f015a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 1 jobs: ['7261211'] suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}'] Timeout 3h running clone.client.0/qa/workunits/rbd/test_lock_fence.sh 1 jobs: ['7261223'] suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_lock_and_fence}'] Failure: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi029 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=12877b4e4694861aa1f9ae444a0b4532908f015a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 1 jobs: ['7261212'] suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests}']