2020-09-13T23:30:15.436 INFO:root:teuthology version: 1.0.0-55bd2dc
2020-09-13T23:30:15.438 INFO:teuthology.results:subset: '1/100000'
2020-09-13T23:30:15.438 INFO:teuthology.results:seed: '4771'
2020-09-14T00:58:29.523 INFO:root:teuthology version: 1.0.0-55bd2dc
2020-09-14T00:58:29.548 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-09-14T06:42:37.597 INFO:teuthology.results:Tests finished! gathering results...
2020-09-14T06:42:53.677 INFO:scrape:Found 151 jobs
2020-09-14T06:42:57.222 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-5231-g3627cd1e5c (3627cd1e5cbc43630daa9774ed7159cdf4dbecf8) pacific (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f58079db880]
,  2: (()+0x276a9a) [0x7f58079dba9a]

2020-09-14T06:42:57.225 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:50.860 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.1 config set osd_debug_deep_scrub_sleep 0
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:{
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-09-14T06:42:57.227 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 10.16 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.229 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2020-09-14T01:02:04.289139+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=4, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.247 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 9.13 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.251 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.511 INFO:tasks.ceph:pgid 8.4 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.252 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 8.11 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 10.12 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.255 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.897 INFO:tasks.ceph:pgid 9.18 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 8.19 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.328 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:04:16.583 INFO:tasks.ceph:pgid 8.2 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.329 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:40.982 INFO:teuthology.orchestra.run:waiting for 300
, 2020-09-14T01:07:41.012 INFO:tasks.ceph.osd.4:Stopped
, 2020-09-14T01:07:41.013 DEBUG:tasks.ceph.osd.5:waiting for process to exit

2020-09-14T06:42:57.329 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:53.102 INFO:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits
, 2020-09-14T01:07:53.200 DEBUG:teuthology.run_tasks:Unwinding manager clock
, 2020-09-14T01:07:53.209 INFO:teuthology.task.clock:Checking final clock skew...

2020-09-14T06:42:57.330 WARNING:scrape:Ignoring malparsed backtrace:     fetch_binaries_for_coredumps(path, rem)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 316, in fetch_binaries_for_coredumps
,     remote_path = remote.sh(['which', dump_program]).rstrip()

2020-09-14T06:42:57.356 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-5231-g3627cd1e5c (3627cd1e5cbc43630daa9774ed7159cdf4dbecf8) pacific (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f58079db880]
,  2: (()+0x276a9a) [0x7f58079dba9a]

2020-09-14T06:42:57.359 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:50.860 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.1 config set osd_debug_deep_scrub_sleep 0
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:{
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-09-14T06:42:57.359 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 10.16 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.361 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2020-09-14T01:02:04.289139+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=4, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.362 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 9.13 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.364 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.511 INFO:tasks.ceph:pgid 8.4 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.364 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 8.11 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 10.12 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.367 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.897 INFO:tasks.ceph:pgid 9.18 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 8.19 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.383 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:04:16.583 INFO:tasks.ceph:pgid 8.2 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.384 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:40.982 INFO:teuthology.orchestra.run:waiting for 300
, 2020-09-14T01:07:41.012 INFO:tasks.ceph.osd.4:Stopped
, 2020-09-14T01:07:41.013 DEBUG:tasks.ceph.osd.5:waiting for process to exit

2020-09-14T06:42:57.385 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:53.102 INFO:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits
, 2020-09-14T01:07:53.200 DEBUG:teuthology.run_tasks:Unwinding manager clock
, 2020-09-14T01:07:53.209 INFO:teuthology.task.clock:Checking final clock skew...

2020-09-14T06:42:57.385 WARNING:scrape:Ignoring malparsed backtrace:     fetch_binaries_for_coredumps(path, rem)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 316, in fetch_binaries_for_coredumps
,     remote_path = remote.sh(['which', dump_program]).rstrip()

2020-09-14T06:42:57.412 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-5231-g3627cd1e5c (3627cd1e5cbc43630daa9774ed7159cdf4dbecf8) pacific (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f58079db880]
,  2: (()+0x276a9a) [0x7f58079dba9a]

2020-09-14T06:42:57.414 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:50.860 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.1 config set osd_debug_deep_scrub_sleep 0
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:{
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-09-14T06:42:57.415 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 10.16 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.417 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2020-09-14T01:02:04.289139+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=4, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.417 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 9.13 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.419 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.511 INFO:tasks.ceph:pgid 8.4 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.420 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 8.11 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 10.12 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.423 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.897 INFO:tasks.ceph:pgid 9.18 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 8.19 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.439 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:04:16.583 INFO:tasks.ceph:pgid 8.2 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.440 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:40.982 INFO:teuthology.orchestra.run:waiting for 300
, 2020-09-14T01:07:41.012 INFO:tasks.ceph.osd.4:Stopped
, 2020-09-14T01:07:41.013 DEBUG:tasks.ceph.osd.5:waiting for process to exit

2020-09-14T06:42:57.441 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:53.102 INFO:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits
, 2020-09-14T01:07:53.200 DEBUG:teuthology.run_tasks:Unwinding manager clock
, 2020-09-14T01:07:53.209 INFO:teuthology.task.clock:Checking final clock skew...

2020-09-14T06:42:57.441 WARNING:scrape:Ignoring malparsed backtrace:     fetch_binaries_for_coredumps(path, rem)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 316, in fetch_binaries_for_coredumps
,     remote_path = remote.sh(['which', dump_program]).rstrip()

2020-09-14T06:42:57.478 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-5231-g3627cd1e5c (3627cd1e5cbc43630daa9774ed7159cdf4dbecf8) pacific (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f58079db880]
,  2: (()+0x276a9a) [0x7f58079dba9a]

2020-09-14T06:42:57.480 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:50.860 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.1 config set osd_debug_deep_scrub_sleep 0
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:{
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-09-14T06:42:57.481 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 10.16 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.483 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2020-09-14T01:02:04.289139+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=4, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.484 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 9.13 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.486 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.511 INFO:tasks.ceph:pgid 8.4 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.486 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 8.11 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 10.12 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.490 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.897 INFO:tasks.ceph:pgid 9.18 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 8.19 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.506 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:04:16.583 INFO:tasks.ceph:pgid 8.2 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.507 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:40.982 INFO:teuthology.orchestra.run:waiting for 300
, 2020-09-14T01:07:41.012 INFO:tasks.ceph.osd.4:Stopped
, 2020-09-14T01:07:41.013 DEBUG:tasks.ceph.osd.5:waiting for process to exit

2020-09-14T06:42:57.508 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:53.102 INFO:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits
, 2020-09-14T01:07:53.200 DEBUG:teuthology.run_tasks:Unwinding manager clock
, 2020-09-14T01:07:53.209 INFO:teuthology.task.clock:Checking final clock skew...

2020-09-14T06:42:57.508 WARNING:scrape:Ignoring malparsed backtrace:     fetch_binaries_for_coredumps(path, rem)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 316, in fetch_binaries_for_coredumps
,     remote_path = remote.sh(['which', dump_program]).rstrip()

2020-09-14T06:42:57.554 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-5231-g3627cd1e5c (3627cd1e5cbc43630daa9774ed7159cdf4dbecf8) pacific (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f58079db880]
,  2: (()+0x276a9a) [0x7f58079dba9a]

2020-09-14T06:42:57.560 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:50.860 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.1 config set osd_debug_deep_scrub_sleep 0
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:{
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-09-14T06:42:57.561 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 10.16 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.564 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2020-09-14T01:02:04.289139+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=4, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.565 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 9.13 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.568 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.511 INFO:tasks.ceph:pgid 8.4 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.569 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 8.11 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 10.12 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.573 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.897 INFO:tasks.ceph:pgid 9.18 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 8.19 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.593 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:04:16.583 INFO:tasks.ceph:pgid 8.2 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.593 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:40.982 INFO:teuthology.orchestra.run:waiting for 300
, 2020-09-14T01:07:41.012 INFO:tasks.ceph.osd.4:Stopped
, 2020-09-14T01:07:41.013 DEBUG:tasks.ceph.osd.5:waiting for process to exit

2020-09-14T06:42:57.594 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:53.102 INFO:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits
, 2020-09-14T01:07:53.200 DEBUG:teuthology.run_tasks:Unwinding manager clock
, 2020-09-14T01:07:53.209 INFO:teuthology.task.clock:Checking final clock skew...

2020-09-14T06:42:57.595 WARNING:scrape:Ignoring malparsed backtrace:     fetch_binaries_for_coredumps(path, rem)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 316, in fetch_binaries_for_coredumps
,     remote_path = remote.sh(['which', dump_program]).rstrip()

2020-09-14T06:42:57.625 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-5231-g3627cd1e5c (3627cd1e5cbc43630daa9774ed7159cdf4dbecf8) pacific (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f58079db880]
,  2: (()+0x276a9a) [0x7f58079dba9a]

2020-09-14T06:42:57.629 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:50.860 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.1 config set osd_debug_deep_scrub_sleep 0
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:{
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-09-14T06:42:57.631 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 10.16 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.633 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2020-09-14T01:02:04.289139+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=4, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.633 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 9.13 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.635 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.511 INFO:tasks.ceph:pgid 8.4 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.636 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 8.11 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 10.12 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.639 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.897 INFO:tasks.ceph:pgid 9.18 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 8.19 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.657 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:04:16.583 INFO:tasks.ceph:pgid 8.2 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.658 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:40.982 INFO:teuthology.orchestra.run:waiting for 300
, 2020-09-14T01:07:41.012 INFO:tasks.ceph.osd.4:Stopped
, 2020-09-14T01:07:41.013 DEBUG:tasks.ceph.osd.5:waiting for process to exit

2020-09-14T06:42:57.659 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:53.102 INFO:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits
, 2020-09-14T01:07:53.200 DEBUG:teuthology.run_tasks:Unwinding manager clock
, 2020-09-14T01:07:53.209 INFO:teuthology.task.clock:Checking final clock skew...

2020-09-14T06:42:57.660 WARNING:scrape:Ignoring malparsed backtrace:     fetch_binaries_for_coredumps(path, rem)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 316, in fetch_binaries_for_coredumps
,     remote_path = remote.sh(['which', dump_program]).rstrip()

2020-09-14T06:42:57.690 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-5231-g3627cd1e5c (3627cd1e5cbc43630daa9774ed7159cdf4dbecf8) pacific (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f58079db880]
,  2: (()+0x276a9a) [0x7f58079dba9a]

2020-09-14T06:42:57.693 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:50.860 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.1 config set osd_debug_deep_scrub_sleep 0
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:{
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-09-14T06:42:57.694 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 10.16 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.696 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2020-09-14T01:02:04.289139+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=4, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.696 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 9.13 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.698 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.511 INFO:tasks.ceph:pgid 8.4 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.699 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 8.11 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 10.12 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.703 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.897 INFO:tasks.ceph:pgid 9.18 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 8.19 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.720 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:04:16.583 INFO:tasks.ceph:pgid 8.2 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.721 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:40.982 INFO:teuthology.orchestra.run:waiting for 300
, 2020-09-14T01:07:41.012 INFO:tasks.ceph.osd.4:Stopped
, 2020-09-14T01:07:41.013 DEBUG:tasks.ceph.osd.5:waiting for process to exit

2020-09-14T06:42:57.721 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:53.102 INFO:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits
, 2020-09-14T01:07:53.200 DEBUG:teuthology.run_tasks:Unwinding manager clock
, 2020-09-14T01:07:53.209 INFO:teuthology.task.clock:Checking final clock skew...

2020-09-14T06:42:57.722 WARNING:scrape:Ignoring malparsed backtrace:     fetch_binaries_for_coredumps(path, rem)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 316, in fetch_binaries_for_coredumps
,     remote_path = remote.sh(['which', dump_program]).rstrip()

2020-09-14T06:42:57.755 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-5231-g3627cd1e5c (3627cd1e5cbc43630daa9774ed7159cdf4dbecf8) pacific (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f58079db880]
,  2: (()+0x276a9a) [0x7f58079dba9a]

2020-09-14T06:42:57.758 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:50.860 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.1 config set osd_debug_deep_scrub_sleep 0
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:{
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-09-14T06:42:57.759 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 10.16 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.761 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2020-09-14T01:02:04.289139+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=4, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.762 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 9.13 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.764 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.511 INFO:tasks.ceph:pgid 8.4 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.765 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 8.11 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 10.12 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.768 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.897 INFO:tasks.ceph:pgid 9.18 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 8.19 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.785 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:04:16.583 INFO:tasks.ceph:pgid 8.2 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.786 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:40.982 INFO:teuthology.orchestra.run:waiting for 300
, 2020-09-14T01:07:41.012 INFO:tasks.ceph.osd.4:Stopped
, 2020-09-14T01:07:41.013 DEBUG:tasks.ceph.osd.5:waiting for process to exit

2020-09-14T06:42:57.787 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:53.102 INFO:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits
, 2020-09-14T01:07:53.200 DEBUG:teuthology.run_tasks:Unwinding manager clock
, 2020-09-14T01:07:53.209 INFO:teuthology.task.clock:Checking final clock skew...

2020-09-14T06:42:57.787 WARNING:scrape:Ignoring malparsed backtrace:     fetch_binaries_for_coredumps(path, rem)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 316, in fetch_binaries_for_coredumps
,     remote_path = remote.sh(['which', dump_program]).rstrip()

2020-09-14T06:42:57.817 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-5231-g3627cd1e5c (3627cd1e5cbc43630daa9774ed7159cdf4dbecf8) pacific (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f58079db880]
,  2: (()+0x276a9a) [0x7f58079dba9a]

2020-09-14T06:42:57.820 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:50.860 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.1 config set osd_debug_deep_scrub_sleep 0
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:{
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-09-14T06:42:57.820 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 10.16 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.823 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2020-09-14T01:02:04.289139+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=4, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.824 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 9.13 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.826 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.511 INFO:tasks.ceph:pgid 8.4 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.827 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 8.11 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 10.12 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.831 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.897 INFO:tasks.ceph:pgid 9.18 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 8.19 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.848 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:04:16.583 INFO:tasks.ceph:pgid 8.2 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.849 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:40.982 INFO:teuthology.orchestra.run:waiting for 300
, 2020-09-14T01:07:41.012 INFO:tasks.ceph.osd.4:Stopped
, 2020-09-14T01:07:41.013 DEBUG:tasks.ceph.osd.5:waiting for process to exit

2020-09-14T06:42:57.850 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:53.102 INFO:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits
, 2020-09-14T01:07:53.200 DEBUG:teuthology.run_tasks:Unwinding manager clock
, 2020-09-14T01:07:53.209 INFO:teuthology.task.clock:Checking final clock skew...

2020-09-14T06:42:57.851 WARNING:scrape:Ignoring malparsed backtrace:     fetch_binaries_for_coredumps(path, rem)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 316, in fetch_binaries_for_coredumps
,     remote_path = remote.sh(['which', dump_program]).rstrip()

2020-09-14T06:42:57.876 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-5231-g3627cd1e5c (3627cd1e5cbc43630daa9774ed7159cdf4dbecf8) pacific (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f58079db880]
,  2: (()+0x276a9a) [0x7f58079dba9a]

2020-09-14T06:42:57.878 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:50.860 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.1 config set osd_debug_deep_scrub_sleep 0
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:{
, 2020-09-14T01:02:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-09-14T06:42:57.879 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.097 INFO:tasks.ceph:pgid 10.16 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.881 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2020-09-14T01:02:04.289139+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=4, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:02:55.123 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.881 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 9.13 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2020-09-14T01:02:08.317758+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=8, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.479 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.883 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:15.511 INFO:tasks.ceph:pgid 8.4 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:15.512 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.884 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2020-09-14T01:02:12.348357+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=12, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 8.11 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.872 INFO:tasks.ceph:pgid 10.12 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.887 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:03:35.897 INFO:tasks.ceph:pgid 9.18 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 8.19 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:03:35.898 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2020-09-14T01:02:10.333958+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=10, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.902 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:04:16.583 INFO:tasks.ceph:pgid 8.2 last_scrub_stamp 2020-09-14T01:02:15.361586+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=15, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2020-09-14T01:02:25.409788+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=25, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)
, 2020-09-14T01:04:16.584 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2020-09-14T01:02:18.370874+0000 time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=18, tm_wday=0, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=9, tm_mday=14, tm_hour=1, tm_min=2, tm_sec=49, tm_wday=0, tm_yday=258, tm_isdst=0)

2020-09-14T06:42:57.903 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:40.982 INFO:teuthology.orchestra.run:waiting for 300
, 2020-09-14T01:07:41.012 INFO:tasks.ceph.osd.4:Stopped
, 2020-09-14T01:07:41.013 DEBUG:tasks.ceph.osd.5:waiting for process to exit

2020-09-14T06:42:57.904 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-14T01:07:53.102 INFO:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits
, 2020-09-14T01:07:53.200 DEBUG:teuthology.run_tasks:Unwinding manager clock
, 2020-09-14T01:07:53.209 INFO:teuthology.task.clock:Checking final clock skew...

2020-09-14T06:42:57.904 WARNING:scrape:Ignoring malparsed backtrace:     fetch_binaries_for_coredumps(path, rem)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 316, in fetch_binaries_for_coredumps
,     remote_path = remote.sh(['which', dump_program]).rstrip()

2020-09-14T06:43:00.134 INFO:scrape:Found 10 distinct failure reasons
2020-09-14T06:43:00.135 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_fsx_prepare.sh) on smithi062 with status 13: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3627cd1e5cbc43630daa9774ed7159cdf4dbecf8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ARGS=\'\' RBD_MIRROR_NOCLEANUP=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_fsx_prepare.sh'
2020-09-14T06:43:00.136 INFO:scrape:1 jobs: ['5416028']
2020-09-14T06:43:00.136 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-fsx-workunit}']
2020-09-14T06:43:00.136 INFO:scrape:
2020-09-14T06:43:00.137 INFO:scrape:Failure: Command failed (workunit test mon/rbd_snaps_ops.sh) on smithi164 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3627cd1e5cbc43630daa9774ed7159cdf4dbecf8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/rbd_snaps_ops.sh'
2020-09-14T06:43:00.137 INFO:scrape:1 jobs: ['5415950']
2020-09-14T06:43:00.137 INFO:scrape:suites: ['objectstore/filestore-xfs', 'openstack', 'rbd/singleton/{all/rbd-vs-unmanaged-snaps', 'supported-random-distro$/{ubuntu_latest}}']
2020-09-14T06:43:00.138 INFO:scrape:
2020-09-14T06:43:00.138 INFO:scrape:Failure: "2020-09-14T00:45:29.005094+0000 osd.0 (osd.0) 3 : cluster [WRN] slow request osd_op(client.10016.0:3953 3.3 3:e30bf49c:test1::rbd_data.271d6c25c57a.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2592768~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e122) initiated 2020-09-14T00:44:58.399660+0000 currently reached pg" in cluster log
2020-09-14T06:43:00.138 INFO:scrape:1 jobs: ['5415974']
2020-09-14T06:43:00.138 INFO:scrape:suites: ['clusters/{fixed-1', 'features/format-1', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'openstack}', 'pool/small-cache-pool', 'rbd/cli_v1/{base/install', 'supported-random-distro$/{rhel_8}', 'workloads/rbd_cli_generic}']
2020-09-14T06:43:00.139 INFO:scrape:
2020-09-14T06:43:00.139 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/qemu_dynamic_features.sh
2020-09-14T06:43:00.140 INFO:scrape:2 jobs: ['5415982', '5415949']
2020-09-14T06:43:00.140 INFO:scrape:suites intersection: ['clusters/{fixed-3', 'openstack}', 'qemu/xfstests', 'rbd/maintenance/{base/install', 'workloads/dynamic_features_no_cache}']
2020-09-14T06:43:00.140 INFO:scrape:suites union: ['clusters/{fixed-3', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zlib', 'openstack}', 'qemu/xfstests', 'rbd/maintenance/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/dynamic_features_no_cache}']
2020-09-14T06:43:00.141 INFO:scrape:
2020-09-14T06:43:00.141 INFO:scrape:Failure: {'Failure object was': {'smithi139.front.sepia.ceph.com': {'msg': 'Failed to download packages: Cannot download Packages/qemu-kvm-block-rbd-2.12.0-88.module_el8.1.0+297+df420408.3.x86_64.rpm: All mirrors were tried', 'results': [], 'invocation': {'module_args': {'name': ['dnf-utils', 'sysstat', 'libedit', 'boost-thread', 'xfsprogs', 'gdisk', 'parted', 'libgcrypt', 'fuse-libs', 'openssl', 'libuuid', 'podman', 'podman-docker', 'attr', 'ant', 'lsof', 'gettext', 'bc', 'xfsdump', 'blktrace', 'usbredir', 'libev-devel', 'valgrind', 'nfs-utils', 'ncurses-devel', 'gcc', 'git', 'python3-nose', 'python3-virtualenv', 'genisoimage', 'qemu-img', 'qemu-kvm-core', 'qemu-kvm-block-rbd', 'libacl-devel', 'dbench', 'autoconf', 'gdb', 'iozone'], 'state': 'present', 'allow_downgrade': False, 'autoremove': False, 'bugfix': False, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'installroot': '/', 'install_repoquery': True, 'install_weak_deps': True, 'security': False, 'skip_broken': False, 'update_cache': False, 'update_only': False, 'validate_certs': True, 'lock_timeout': 30, 'conf_file': 'None', 'disable_excludes': 'None', 'download_dir': 'None', 'list': 'None', 'releasever': 'None'}}, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '/')"}
2020-09-14T06:43:00.142 INFO:scrape:1 jobs: ['5415978']
2020-09-14T06:43:00.142 INFO:scrape:suites: ['objectstore/bluestore-comp-zlib', 'openstack', 'rbd/singleton/{all/read-flags-no-cache', 'supported-random-distro$/{centos_8}}']
2020-09-14T06:43:00.142 INFO:scrape:
2020-09-14T06:43:00.143 INFO:scrape:Failure: Command failed (workunit test rbd/qemu_dynamic_features.sh) on smithi075 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3627cd1e5cbc43630daa9774ed7159cdf4dbecf8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 IMAGE_NAME=client.0.1-clone adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/qemu_dynamic_features.sh'
2020-09-14T06:43:00.143 INFO:scrape:2 jobs: ['5415969', '5415926']
2020-09-14T06:43:00.143 INFO:scrape:suites intersection: ['clusters/{fixed-3', 'openstack}', 'qemu/xfstests', 'rbd/maintenance/{base/install', 'workloads/dynamic_features}']
2020-09-14T06:43:00.144 INFO:scrape:suites union: ['clusters/{fixed-3', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-stupid', 'openstack}', 'qemu/xfstests', 'rbd/maintenance/{base/install', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/dynamic_features}']
2020-09-14T06:43:00.144 INFO:scrape:
2020-09-14T06:43:00.144 INFO:scrape:Failure: "2020-09-14T00:02:26.823648+0000 mon.a (mon.0) 44 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log
2020-09-14T06:43:00.144 INFO:scrape:1 jobs: ['5415914']
2020-09-14T06:43:00.145 INFO:scrape:suites: ['clusters/{fixed-3', 'config/permit-partial-discard', 'min-compat-client/octopus', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'openstack}', 'pool/small-cache-pool', 'rbd/librbd/{cache/writethrough', 'supported-random-distro$/{ubuntu_latest}', 'workloads/fsx}']
2020-09-14T06:43:00.145 INFO:scrape:
2020-09-14T06:43:00.146 INFO:scrape:Failure: Command failed (workunit test rbd/rbd-nbd.sh) on smithi172 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3627cd1e5cbc43630daa9774ed7159cdf4dbecf8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh'
2020-09-14T06:43:00.148 INFO:scrape:1 jobs: ['5416036']
2020-09-14T06:43:00.149 INFO:scrape:suites: ['cluster/{fixed-3', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'openstack}', 'rbd/nbd/{base/install', 'supported-random-distro$/{centos_8}', 'thrashers/cache', 'thrashosds-health', 'workloads/rbd_nbd}']
2020-09-14T06:43:00.149 INFO:scrape:
2020-09-14T06:43:00.150 INFO:scrape:Failure: Command failed (workunit test rbd/test_librbd.sh) on smithi194 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3627cd1e5cbc43630daa9774ed7159cdf4dbecf8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 RBD_FEATURES=125 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh'
2020-09-14T06:43:00.150 INFO:scrape:1 jobs: ['5416019']
2020-09-14T06:43:00.151 INFO:scrape:suites: ['clusters/{fixed-3', 'config/copy-on-read', 'min-compat-client/default', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'openstack}', 'pool/replicated-data-pool', 'rbd/librbd/{cache/writeback', 'supported-random-distro$/{centos_8}', 'workloads/c_api_tests_with_journaling}']
2020-09-14T06:43:00.151 INFO:scrape:
2020-09-14T06:43:00.151 INFO:scrape:Failure: Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx --cluster cluster1 --id mirror.0 -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 1 -w 1 -t 1 -h 1 -l 250000000 -S 0 -N 6000 -k mirror image_cluster1.client.mirror.0'
2020-09-14T06:43:00.152 INFO:scrape:1 jobs: ['5415930']
2020-09-14T06:43:00.152 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{rhel_8}', 'workloads/rbd-mirror-fsx-workunit}']
2020-09-14T06:43:00.152 INFO:scrape:
2020-09-14T06:43:00.159 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/
failed:  11
dead:    1
running: 0
waiting: 0
queued:  0
passed:  139


Fail
=================================================================
[5415914]  rbd/librbd/{cache/writethrough clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-snappy pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/fsx}
-----------------------------------------------------------------
time:   00:22:27
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415914/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415914/

    "2020-09-14T00:02:26.823648+0000 mon.a (mon.0) 44 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)"
    in cluster log

[5415926]  rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-stupid qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/dynamic_features}
-----------------------------------------------------------------
time:   06:35:31
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415926/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415926/
sentry: https://sentry.ceph.com/sentry/sepia/?query=e4fde444c08c4f789c9dbeddd1b4c901

    Command failed (workunit test rbd/qemu_dynamic_features.sh) on smithi051
    with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd
    -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=3627cd1e5cbc43630daa9774ed7159cdf4dbecf8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 IMAGE_NAME=client.0.1-clone
    adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout
    3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/qemu_dynamic_featu
    res.sh'

[5415930]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-fsx-workunit}
-----------------------------------------------------------------
time:   00:57:30
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415930/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415930/
sentry: https://sentry.ceph.com/sentry/sepia/?query=32198c9f75bd42caab758bee011580e4

    Command crashed: 'adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx --cluster
    cluster1 --id mirror.0 -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r
    1 -w 1 -t 1 -h 1 -l 250000000 -S 0 -N 6000 -k mirror
    image_cluster1.client.mirror.0'

[5415949]  rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-bitmap qemu/xfstests supported-random-distro$/{centos_8} workloads/dynamic_features_no_cache}
-----------------------------------------------------------------
time:   03:08:48
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415949/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415949/
sentry: https://sentry.ceph.com/sentry/sepia/?query=e58d759f449a4e24a34732df18836ac4

    Command failed (workunit test rbd/qemu_dynamic_features.sh) on smithi167
    with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=3627cd1e5cbc43630daa9774ed7159cdf4dbecf8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 IMAGE_NAME=client.0.1-clone
    adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout
    3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/qemu_dynamic_featu
    res.sh'

[5415950]  rbd/singleton/{all/rbd-vs-unmanaged-snaps objectstore/filestore-xfs openstack supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:05:47
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415950/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415950/
sentry: https://sentry.ceph.com/sentry/sepia/?query=22d8f1a12b9744439906ed3a4697789d

    Command failed (workunit test mon/rbd_snaps_ops.sh) on smithi164 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=3627cd1e5cbc43630daa9774ed7159cdf4dbecf8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/rbd_snaps_ops.sh'

[5415969]  rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-comp-snappy qemu/xfstests supported-random-distro$/{rhel_8} workloads/dynamic_features}
-----------------------------------------------------------------
time:   00:12:53
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415969/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415969/
sentry: https://sentry.ceph.com/sentry/sepia/?query=2dccc6cd535e45b6ac67edd34e828736

    Command failed (workunit test rbd/qemu_dynamic_features.sh) on smithi075
    with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd
    -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=3627cd1e5cbc43630daa9774ed7159cdf4dbecf8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 IMAGE_NAME=client.0.1-clone
    adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout
    3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/qemu_dynamic_featu
    res.sh'

[5415974]  rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-bitmap pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/rbd_cli_generic}
-----------------------------------------------------------------
time:   00:25:15
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415974/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415974/

    "2020-09-14T00:45:29.005094+0000 osd.0 (osd.0) 3 : cluster [WRN] slow
    request osd_op(client.10016.0:3953 3.3
    3:e30bf49c:test1::rbd_data.271d6c25c57a.0000000000000000:head [set-alloc-
    hint object_size 4194304 write_size 4194304,write 2592768~4096
    [fadvise_random] in=4096b] snapc 0=[] RETRY=4
    ondisk+retry+write+known_if_redirected e122) initiated
    2020-09-14T00:44:58.399660+0000 currently reached pg" in cluster log

[5415982]  rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-comp-zlib qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/dynamic_features_no_cache}
-----------------------------------------------------------------
time:   05:45:58
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415982/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415982/
sentry: https://sentry.ceph.com/sentry/sepia/?query=f9d6ef5be3844e258501f74d798df764

    Command failed (workunit test rbd/qemu_dynamic_features.sh) on smithi067
    with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=3627cd1e5cbc43630daa9774ed7159cdf4dbecf8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 IMAGE_NAME=client.0.1-clone
    adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout
    3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/qemu_dynamic_featu
    res.sh'

[5416019]  rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-bitmap pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/c_api_tests_with_journaling}
-----------------------------------------------------------------
time:   00:12:34
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416019/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416019/
sentry: https://sentry.ceph.com/sentry/sepia/?query=41cbce4573564cb8bd25c6be0b0d9b99

    Command failed (workunit test rbd/test_librbd.sh) on smithi194 with status
    134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=3627cd1e5cbc43630daa9774ed7159cdf4dbecf8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 RBD_FEATURES=125 adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh'

[5416028]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-fsx-workunit}
-----------------------------------------------------------------
time:   00:13:16
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416028/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416028/
sentry: https://sentry.ceph.com/sentry/sepia/?query=4a2e0aa412ee49e59062aa405644cfbf

    Command failed (workunit test rbd/rbd_mirror_fsx_prepare.sh) on smithi062
    with status 13: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=3627cd1e5cbc43630daa9774ed7159cdf4dbecf8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_NOCLEANUP=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_fsx_prepare.sh'

[5416036]  rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} thrashers/cache thrashosds-health workloads/rbd_nbd}
-----------------------------------------------------------------
time:   00:08:16
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416036/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416036/
sentry: https://sentry.ceph.com/sentry/sepia/?query=031510d3e8eb40a5b8642626a41a7a2c

    Command failed (workunit test rbd/rbd-nbd.sh) on smithi172 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=3627cd1e5cbc43630daa9774ed7159cdf4dbecf8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh'



Dead
=================================================================
[5415978]  rbd/singleton/{all/read-flags-no-cache objectstore/bluestore-comp-zlib openstack supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:01:25
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415978/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415978/
sentry: https://sentry.ceph.com/sentry/sepia/?query=8eade29221654a8b964bb3285e380cc2

    {'Failure object was': {'smithi139.front.sepia.ceph.com': {'msg': 'Failed
    to download packages: Cannot download Packages/qemu-kvm-block-
    rbd-2.12.0-88.module_el8.1.0+297+df420408.3.x86_64.rpm: All mirrors were
    tried', 'results': [], 'invocation': {'module_args': {'name': ['dnf-utils',
    'sysstat', 'libedit', 'boost-thread', 'xfsprogs', 'gdisk', 'parted',
    'libgcrypt', 'fuse-libs', 'openssl', 'libuuid', 'podman', 'podman-docker',
    'attr', 'ant', 'lsof', 'gettext', 'bc', 'xfsdump', 'blktrace', 'usbredir',
    'libev-devel', 'valgrind', 'nfs-utils', 'ncurses-devel', 'gcc', 'git',
    'python3-nose', 'python3-virtualenv', 'genisoimage', 'qemu-img', 'qemu-kvm-
    core', 'qemu-kvm-block-rbd', 'libacl-devel', 'dbench', 'autoconf', 'gdb',
    'iozone'], 'state': 'present', 'allow_downgrade': False, 'autoremove':
    False, 'bugfix': False, 'disable_gpg_check': False, 'disable_plugin': [],
    'disablerepo': [], 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'installroot': '/', 'install_repoquery':
    True, 'install_weak_deps': True, 'security': False, 'skip_broken': False,
    'update_cache': False, 'update_only': False, 'validate_certs': True,
    'lock_timeout': 30, 'conf_file': 'None', 'disable_excludes': 'None',
    'download_dir': 'None', 'list': 'None', 'releasever': 'None'}},
    '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call
    last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-
    ansible_master/callback_plugins/failure_log.py", line 44, in log_failure
    log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com
    _git_teuthology_master/virtualenv/lib/python3.6/site-
    packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data],
    stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com
    _git_teuthology_master/virtualenv/lib/python3.6/site-
    packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data)
    File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/l
    ib/python3.6/site-packages/yaml/representer.py", line 27, in represent node
    = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_te
    uthology_master/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 48, in represent_data node =
    self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s
    rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 207, in represent_dict return
    self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo
    rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 118, in represent_mapping node_value =
    self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi
    t_teuthology_master/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 48, in represent_data node =
    self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s
    rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 207, in represent_dict return
    self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo
    rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 118, in represent_mapping node_value =
    self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi
    t_teuthology_master/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 48, in represent_data node =
    self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s
    rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 207, in represent_dict return
    self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo
    rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 118, in represent_mapping node_value =
    self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi
    t_teuthology_master/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 48, in represent_data node =
    self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s
    rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 207, in represent_dict return
    self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo
    rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 118, in represent_mapping node_value =
    self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi
    t_teuthology_master/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 58, in represent_data node =
    self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce
    ph.com_git_teuthology_master/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 231, in represent_undefined raise
    RepresenterError("cannot represent an object", data)',
    'yaml.representer.RepresenterError': "('cannot represent an object', '/')"}



Pass
=================================================================
[5415891] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} workloads/rbd-mirror-ha-workunit}
time:   00:18:19
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415891/

[5415892] rbd/basic/{base/install cachepool/none clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} tasks/rbd_lock_and_fence}
time:   00:06:40
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415892/

[5415893] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-lz4 validator/memcheck workloads/c_api_tests_with_journaling}
time:   01:20:01
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415893/

[5415894] rbd/cli/{base/install clusters/{fixed-1 openstack} features/journaling msgr-failures/few objectstore/bluestore-comp-lz4 pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/rbd_cli_groups}
time:   00:06:54
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415894/

[5415895] rbd/singleton/{all/merge_diff objectstore/bluestore-comp-lz4 openstack supported-random-distro$/{rhel_8}}
time:   00:13:19
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415895/

[5415896] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/none min-compat-client/default msgr-failures/few objectstore/bluestore-comp-lz4 pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/c_api_tests_with_journaling}
time:   00:38:02
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415896/

[5415897] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/bluestore-comp-lz4 pool/ec-cache-pool supported-random-distro$/{rhel_8} workloads/qemu_fsstress}
time:   00:34:07
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415897/

[5415898] rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-comp-zlib pool/none supported-random-distro$/{centos_8} workloads/rbd_cli_generic}
time:   00:19:30
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415898/

[5415899] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} thrashers/cache thrashosds-health workloads/rbd_fsx_nbd}
time:   00:19:30
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415899/

[5415900] rbd/immutable-object-cache/{clusters/{fix-2 openstack} pool/ceph_and_immutable_object_cache supported-random-distro$/{centos_8} workloads/c_api_tests_with_defaults}
time:   00:20:49
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415900/

[5415901] rbd/singleton-bluestore/{all/issue-20295 objectstore/bluestore-bitmap openstack supported-random-distro$/{centos_8}}
time:   00:18:32
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415901/

[5415902] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-workunit}
time:   00:34:49
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415902/

[5415903] rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_api_tests_copy_on_read}
time:   00:27:17
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415903/

[5415904] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-hybrid qemu/xfstests supported-random-distro$/{centos_8} workloads/dynamic_features_no_cache}
time:   04:52:43
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415904/

[5415905] rbd/singleton/{all/permissions objectstore/bluestore-comp-snappy openstack supported-random-distro$/{rhel_8}}
time:   00:11:14
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415905/

[5415906] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
time:   00:14:41
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415906/

[5415907] rbd/basic/{base/install cachepool/small clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/rbd_python_api_tests_old_format}
time:   00:17:29
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415907/

[5415908] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-snappy validator/memcheck workloads/fsx}
time:   00:15:19
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415908/

[5415909] rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-comp-zstd pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_cli_import_export}
time:   00:06:56
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415909/

[5415910] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_nbd}
time:   00:07:58
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415910/

[5415911] rbd/cli/{base/install clusters/{fixed-1 openstack} features/layering msgr-failures/few objectstore/bluestore-comp-snappy pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_cli_import_export}
time:   00:06:51
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415911/

[5415912] rbd/singleton/{all/qemu-iotests-no-cache objectstore/bluestore-comp-zlib openstack supported-random-distro$/{ubuntu_latest}}
time:   00:09:50
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415912/

[5415913] rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} thrashers/cache thrashosds-health workloads/rbd_api_tests_journaling}
time:   00:54:18
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415913/

[5415915] rbd/qemu/{cache/writethrough clusters/{fixed-3 openstack} features/readbalance msgr-failures/few objectstore/bluestore-comp-snappy pool/ec-data-pool supported-random-distro$/{rhel_8} workloads/qemu_xfstests}
time:   06:20:38
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415915/

[5415916] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-low-osd-mem-target qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/rebuild_object_map}
time:   04:01:29
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415916/

[5415917] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-stress-workunit}
time:   00:22:55
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415917/

[5415918] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-fast-diff}
time:   00:14:51
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415918/

[5415919] rbd/singleton/{all/qemu-iotests-writearound objectstore/bluestore-comp-zstd openstack supported-random-distro$/{rhel_8}}
time:   00:11:15
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415919/

[5415920] rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_api_tests_no_locking}
time:   00:28:13
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415920/

[5415921] rbd/basic/{base/install cachepool/none clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/rbd_api_tests_old_format}
time:   00:19:02
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415921/

[5415922] rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-hybrid pool/none supported-random-distro$/{rhel_8} workloads/rbd_cli_generic}
time:   00:21:33
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415922/

[5415923] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} thrashers/cache thrashosds-health workloads/rbd_fsx_nbd}
time:   00:14:04
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415923/

[5415924] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-zlib validator/memcheck workloads/python_api_tests}
time:   00:28:10
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415924/

[5415925] rbd/cli/{base/install clusters/{fixed-1 openstack} features/defaults msgr-failures/few objectstore/bluestore-comp-zlib pool/ec-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_generic}
time:   00:21:38
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415925/

[5415927] rbd/singleton/{all/qemu-iotests-writeback objectstore/bluestore-hybrid openstack supported-random-distro$/{centos_8}}
time:   00:07:25
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415927/

[5415928] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-comp-zlib pool/ec-data-pool supported-random-distro$/{centos_8} workloads/python_api_tests}
time:   00:17:32
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415928/

[5415929] rbd/qemu/{cache/none clusters/{fixed-3 openstack} features/defaults msgr-failures/few objectstore/bluestore-comp-zlib pool/none supported-random-distro$/{centos_8} workloads/qemu_bonnie}
time:   00:23:41
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415929/

[5415931] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-journaling}
time:   00:16:34
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415931/

[5415932] rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} thrashers/cache thrashosds-health workloads/rbd_fsx_cache_writearound}
time:   00:15:26
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415932/

[5415933] rbd/singleton/{all/qemu-iotests-writethrough objectstore/bluestore-low-osd-mem-target openstack supported-random-distro$/{centos_8}}
time:   00:07:09
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415933/

[5415934] rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/rbd_cli_import_export}
time:   00:11:11
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415934/

[5415935] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd}
time:   00:08:01
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415935/

[5415936] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/filestore-xfs qemu/xfstests supported-random-distro$/{centos_8} workloads/dynamic_features_no_cache}
time:   01:41:10
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415936/

[5415937] rbd/basic/{base/install cachepool/small clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} tasks/rbd_cls_tests}
time:   00:12:17
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415937/

[5415938] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-zstd validator/memcheck workloads/python_api_tests_with_defaults}
time:   00:28:31
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415938/

[5415939] rbd/cli/{base/install clusters/{fixed-1 openstack} features/journaling msgr-failures/few objectstore/bluestore-comp-zstd pool/none supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_groups}
time:   00:06:05
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415939/

[5415940] rbd/singleton/{all/qos objectstore/bluestore-stupid openstack supported-random-distro$/{centos_8}}
time:   00:38:52
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415940/

[5415941] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-minimum}
time:   00:15:41
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415941/

[5415942] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-workunit}
time:   00:29:36
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415942/

[5415943] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-zstd pool/none supported-random-distro$/{centos_8} workloads/python_api_tests_with_defaults}
time:   00:17:58
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415943/

[5415944] rbd/qemu/{cache/writearound clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/bluestore-comp-zstd pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/qemu_fsstress}
time:   00:23:13
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415944/

[5415945] rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/rbd_fsx_cache_writeback}
time:   00:14:46
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415945/

[5415946] rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-stupid pool/none supported-random-distro$/{rhel_8} workloads/rbd_cli_generic}
time:   00:22:50
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415946/

[5415947] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8} thrashers/cache thrashosds-health workloads/rbd_fsx_nbd}
time:   00:13:22
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415947/

[5415948] rbd/immutable-object-cache/{clusters/{fix-2 openstack} pool/ceph_and_immutable_object_cache supported-random-distro$/{centos_8} workloads/fio_on_immutable_object_cache}
time:   00:24:10
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415948/

[5415951] rbd/basic/{base/install cachepool/none clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/rbd_lock_and_fence}
time:   00:06:40
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415951/

[5415952] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-hybrid validator/memcheck workloads/python_api_tests_with_journaling}
time:   00:29:52
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415952/

[5415953] rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} thrashers/cache thrashosds-health workloads/rbd_fsx_cache_writethrough}
time:   00:13:05
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415953/

[5415954] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{rhel_8} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus}
time:   00:19:29
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415954/

[5415955] rbd/cli/{base/install clusters/{fixed-1 openstack} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_import_export}
time:   00:07:01
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415955/

[5415956] rbd/singleton/{all/rbd_mirror objectstore/bluestore-bitmap openstack supported-random-distro$/{centos_8}}
time:   00:41:26
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415956/

[5415957] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-stress-workunit}
time:   00:25:46
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415957/

[5415958] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/bluestore-hybrid pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/python_api_tests_with_journaling}
time:   00:18:49
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415958/

[5415959] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/readbalance msgr-failures/few objectstore/bluestore-hybrid pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/qemu_xfstests}
time:   02:44:58
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415959/

[5415960] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-comp-lz4 qemu/xfstests supported-random-distro$/{rhel_8} workloads/rebuild_object_map}
time:   03:22:26
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415960/

[5415961] rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/rbd_cli_import_export}
time:   00:10:55
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415961/

[5415962] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/rbd_nbd}
time:   00:12:29
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415962/

[5415963] rbd/singleton/{all/rbd_tasks objectstore/bluestore-comp-lz4 openstack supported-random-distro$/{ubuntu_latest}}
time:   00:07:00
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415963/

[5415964] rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_copy_on_read}
time:   00:10:34
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415964/

[5415965] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} workloads/rbd-mirror-workunit-config-key}
time:   00:24:42
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415965/

[5415966] rbd/basic/{base/install cachepool/small clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/rbd_python_api_tests_old_format}
time:   00:17:38
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415966/

[5415967] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-low-osd-mem-target validator/memcheck workloads/rbd_mirror}
time:   00:52:51
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415967/

[5415968] rbd/cli/{base/install clusters/{fixed-1 openstack} features/defaults msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_generic}
time:   00:20:05
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415968/

[5415970] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-fsx-workunit}
time:   01:26:05
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415970/

[5415971] rbd/singleton/{all/rbdmap_RBDMAPFILE objectstore/bluestore-comp-snappy openstack supported-random-distro$/{centos_8}}
time:   00:05:28
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415971/

[5415972] rbd/librbd/{cache/writethrough clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/rbd_fio}
time:   01:04:18
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415972/

[5415973] rbd/qemu/{cache/writethrough clusters/{fixed-3 openstack} features/defaults msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/ec-cache-pool supported-random-distro$/{ubuntu_latest} workloads/qemu_bonnie}
time:   00:24:35
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415973/

[5415975] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_nbd}
time:   00:18:42
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415975/

[5415976] rbd/singleton-bluestore/{all/issue-20295 objectstore/bluestore-comp-snappy openstack supported-random-distro$/{ubuntu_latest}}
time:   00:20:04
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415976/

[5415977] rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} thrashers/cache thrashosds-health workloads/rbd_fsx_deep_copy}
time:   00:14:17
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415977/

[5415979] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-min-compat-client-octopus}
time:   00:19:58
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415979/

[5415980] rbd/basic/{base/install cachepool/none clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/rbd_api_tests_old_format}
time:   00:21:41
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415980/

[5415981] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-stupid validator/memcheck workloads/c_api_tests}
time:   00:41:34
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415981/

[5415983] rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/rbd_fsx_journal}
time:   00:30:41
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415983/

[5415984] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-workunit}
time:   00:33:48
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415984/

[5415985] rbd/cli/{base/install clusters/{fixed-1 openstack} features/journaling msgr-failures/few objectstore/bluestore-stupid pool/ec-data-pool supported-random-distro$/{rhel_8} workloads/rbd_cli_groups}
time:   00:11:33
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415985/

[5415986] rbd/singleton/{all/read-flags-writeback objectstore/bluestore-comp-zstd openstack supported-random-distro$/{ubuntu_latest}}
time:   00:05:55
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415986/

[5415987] rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-comp-lz4 pool/none supported-random-distro$/{centos_8} workloads/rbd_cli_import_export}
time:   00:07:20
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415987/

[5415988] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} thrashers/cache thrashosds-health workloads/rbd_nbd}
time:   00:12:39
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415988/

[5415989] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/none min-compat-client/default msgr-failures/few objectstore/bluestore-stupid pool/ec-data-pool supported-random-distro$/{centos_8} workloads/c_api_tests}
time:   00:22:02
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415989/

[5415990] rbd/qemu/{cache/none clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/bluestore-stupid pool/ec-data-pool supported-random-distro$/{centos_8} workloads/qemu_fsstress}
time:   00:23:14
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415990/

[5415991] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} workloads/rbd-mirror-workunit-policy-none}
time:   00:19:55
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415991/

[5415992] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-comp-zstd qemu/xfstests supported-random-distro$/{rhel_8} workloads/rebuild_object_map}
time:   03:50:58
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415992/

[5415993] rbd/singleton/{all/read-flags-writethrough objectstore/bluestore-hybrid openstack supported-random-distro$/{centos_8}}
time:   00:06:37
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415993/

[5415994] rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} thrashers/cache thrashosds-health workloads/rbd_fsx_nocache}
time:   00:12:03
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415994/

[5415995] rbd/basic/{base/install cachepool/small clusters/{fixed-1 openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_8} tasks/rbd_cls_tests}
time:   00:08:07
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415995/

[5415996] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/filestore-xfs validator/memcheck workloads/c_api_tests_with_defaults}
time:   00:41:48
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415996/

[5415997] rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-comp-snappy pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_generic}
time:   00:21:15
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415997/

[5415998] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/rbd_fsx_nbd}
time:   00:15:17
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415998/

[5415999] rbd/immutable-object-cache/{clusters/{fix-2 openstack} pool/ceph_and_immutable_object_cache supported-random-distro$/{ubuntu_latest} workloads/qemu_on_immutable_object_cache_and_thrash}
time:   01:39:38
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5415999/

[5416000] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-workunit}
time:   00:29:56
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416000/

[5416001] rbd/cli/{base/install clusters/{fixed-1 openstack} features/layering msgr-failures/few objectstore/filestore-xfs pool/none supported-random-distro$/{rhel_8} workloads/rbd_cli_import_export}
time:   00:11:22
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416001/

[5416002] rbd/singleton/{all/snap-diff objectstore/bluestore-low-osd-mem-target openstack supported-random-distro$/{ubuntu_latest}}
time:   00:05:43
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416002/

[5416003] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/filestore-xfs pool/none supported-random-distro$/{centos_8} workloads/c_api_tests_with_defaults}
time:   00:21:45
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416003/

[5416004] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-policy-simple}
time:   00:19:52
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416004/

[5416005] rbd/qemu/{cache/writearound clusters/{fixed-3 openstack} features/readbalance msgr-failures/few objectstore/filestore-xfs pool/none supported-random-distro$/{rhel_8} workloads/qemu_xfstests}
time:   01:10:37
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416005/

[5416006] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-hybrid qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/dynamic_features}
time:   04:47:42
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416006/

[5416007] rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_rate_limit}
time:   00:12:36
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416007/

[5416008] rbd/singleton/{all/verify_pool objectstore/bluestore-stupid openstack supported-random-distro$/{ubuntu_latest}}
time:   00:05:39
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416008/

[5416009] rbd/basic/{base/install cachepool/none clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/rbd_lock_and_fence}
time:   00:06:21
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416009/

[5416010] rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-comp-zlib pool/none supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_import_export}
time:   00:06:14
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416010/

[5416011] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} thrashers/cache thrashosds-health workloads/rbd_nbd}
time:   00:08:55
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416011/

[5416012] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-bitmap validator/memcheck workloads/c_api_tests_with_journaling}
time:   01:27:39
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416012/

[5416013] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-stress-workunit}
time:   00:21:31
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416013/

[5416014] rbd/cli/{base/install clusters/{fixed-1 openstack} features/defaults msgr-failures/few objectstore/bluestore-bitmap pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_generic}
time:   00:20:45
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416014/

[5416015] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-low-osd-mem-target qemu/xfstests supported-random-distro$/{rhel_8} workloads/dynamic_features_no_cache}
time:   04:26:03
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416015/

[5416016] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} workloads/rbd-mirror-bootstrap-workunit}
time:   00:09:12
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416016/

[5416017] rbd/singleton/{all/admin_socket objectstore/bluestore-stupid openstack supported-random-distro$/{centos_8}}
time:   00:06:14
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416017/

[5416018] rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_8} thrashers/cache thrashosds-health workloads/journal}
time:   00:07:55
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416018/

[5416020] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/defaults msgr-failures/few objectstore/bluestore-bitmap pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/qemu_bonnie}
time:   00:16:11
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416020/

[5416021] rbd/singleton/{all/formatted-output objectstore/filestore-xfs openstack supported-random-distro$/{centos_8}}
time:   00:06:42
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416021/

[5416022] rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-comp-zstd pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_generic}
time:   00:21:22
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416022/

[5416023] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_fsx_nbd}
time:   00:13:35
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416023/

[5416024] rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/rbd_api_tests}
time:   00:32:14
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416024/

[5416025] rbd/basic/{base/install cachepool/small clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} tasks/rbd_python_api_tests_old_format}
time:   00:16:50
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416025/

[5416026] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-lz4 validator/memcheck workloads/fsx}
time:   00:18:19
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416026/

[5416027] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-stupid qemu/xfstests supported-random-distro$/{rhel_8} workloads/rebuild_object_map}
time:   04:58:32
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416027/

[5416029] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-ha-workunit}
time:   00:14:49
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416029/

[5416030] rbd/cli/{base/install clusters/{fixed-1 openstack} features/journaling msgr-failures/few objectstore/bluestore-comp-lz4 pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/rbd_cli_groups}
time:   00:11:08
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416030/

[5416031] rbd/singleton/{all/merge_diff objectstore/bluestore-bitmap openstack supported-random-distro$/{rhel_8}}
time:   00:13:54
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416031/

[5416032] rbd/librbd/{cache/writethrough clusters/{fixed-3 openstack} config/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-lz4 pool/small-cache-pool supported-random-distro$/{centos_8} workloads/fsx}
time:   00:15:36
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416032/

[5416033] rbd/qemu/{cache/writethrough clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/bluestore-comp-lz4 pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/qemu_fsstress}
time:   00:31:31
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416033/

[5416034] rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} thrashers/cache thrashosds-health workloads/rbd_api_tests_copy_on_read}
time:   00:35:52
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416034/

[5416035] rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-hybrid pool/none supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_import_export}
time:   00:06:13
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416035/

[5416037] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/filestore-xfs qemu/xfstests supported-random-distro$/{rhel_8} workloads/dynamic_features}
time:   01:27:28
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416037/

[5416038] rbd/singleton/{all/permissions objectstore/bluestore-comp-lz4 openstack supported-random-distro$/{rhel_8}}
time:   00:11:41
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416038/

[5416039] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
time:   00:14:49
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416039/

[5416040] rbd/basic/{base/install cachepool/none clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/rbd_api_tests_old_format}
time:   00:22:35
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416040/

[5416041] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-snappy validator/memcheck workloads/python_api_tests}
time:   00:29:59
info:   http://pulpito.ceph.com/teuthology-2020-09-08_02:01:03-rbd-master-distro-basic-smithi/5416041/