2020-10-01T10:11:40.653 INFO:root:teuthology version: 1.0.0-75888fc 2020-10-01T10:11:40.654 INFO:teuthology.results:seed: '534' 2020-10-01T10:17:33.892 INFO:root:teuthology version: 1.0.0-75888fc 2020-10-01T10:17:33.895 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-10-01T11:24:35.048 INFO:teuthology.results:Tests finished! gathering results... 2020-10-01T11:24:37.181 INFO:scrape:Found 20 jobs 2020-10-01T11:24:37.288 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-5799-g5c3ba95b (5c3ba95b4d1208868db1d00d1ba06ed086e1b268) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f8f330b54f8] , 2: () 2020-10-01T11:24:37.326 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:47:45.176 INFO:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.1 config set osd_debug_deep_scrub_sleep 0 , 2020-10-01T10:47:45.319 INFO:teuthology.orchestra.run.smithi121.stdout:{ , 2020-10-01T10:47:45.319 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-10-01T11:24:37.327 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:47:48.757 INFO:tasks.ceph:pgid 10.16 last_scrub_stamp 2020-10-01T10:47:19.514837+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=19, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:47:48.757 INFO:tasks.ceph:pgid 9.15 last_scrub_stamp 2020-10-01T10:47:12.471296+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=12, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:47:48.758 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2020-10-01T10:47:02.501999+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=2, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.329 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:47:48.782 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2020-10-01T10:47:04.511349+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=4, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:47:48.782 INFO:tasks.ceph:pgid 7.9 last_scrub_stamp 2020-10-01T10:47:06.525170+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=6, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:47:48.782 INFO:tasks.ceph:pgid 8.6 last_scrub_stamp 2020-10-01T10:47:09.459489+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=9, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.330 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:48:09.150 INFO:tasks.ceph:pgid 8.12 last_scrub_stamp 2020-10-01T10:47:09.459489+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=9, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:09.151 INFO:tasks.ceph:pgid 10.13 last_scrub_stamp 2020-10-01T10:47:19.514837+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=19, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:09.151 INFO:tasks.ceph:pgid 9.10 last_scrub_stamp 2020-10-01T10:47:12.471296+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=12, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.332 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:48:09.181 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2020-10-01T10:47:04.511349+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=4, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:09.181 INFO:tasks.ceph:pgid 7.f last_scrub_stamp 2020-10-01T10:47:06.525170+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=6, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:09.182 INFO:tasks.ceph:pgid 8.0 last_scrub_stamp 2020-10-01T10:47:09.459489+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=9, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.334 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:48:29.550 INFO:tasks.ceph:pgid 7.14 last_scrub_stamp 2020-10-01T10:47:06.525170+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=6, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:29.550 INFO:tasks.ceph:pgid 10.f last_scrub_stamp 2020-10-01T10:47:19.514837+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=19, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:29.550 INFO:tasks.ceph:pgid 9.c last_scrub_stamp 2020-10-01T10:47:12.471296+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=12, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.336 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:48:49.936 INFO:tasks.ceph:pgid 8.16 last_scrub_stamp 2020-10-01T10:47:09.459489+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=9, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:49.936 INFO:tasks.ceph:pgid 10.17 last_scrub_stamp 2020-10-01T10:47:19.514837+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=19, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:49.937 INFO:tasks.ceph:pgid 9.14 last_scrub_stamp 2020-10-01T10:47:12.471296+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=12, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.375 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:49:10.305 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-01T10:49:30.306 INFO:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-10-01T10:49:30.619 INFO:teuthology.orchestra.run.smithi121.stdout: 2020-10-01T11:24:37.375 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:52:14.638 INFO:tasks.ceph.mon.a:Stopped , 2020-10-01T10:52:14.638 DEBUG:tasks.ceph.mon.c:waiting for process to exit , 2020-10-01T10:52:14.639 INFO:teuthology.orchestra.run:waiting for 300 2020-10-01T11:24:37.376 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:52:27.001 INFO:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , bash: ntpq: command not found , 2020-10-01T10:52:27.020 INFO:teuthology.orchestra.run.smithi162.stdout:210 Number of sources = 4 2020-10-01T11:24:37.376 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:52:58.724 INFO:teuthology.orchestra.run.smithi035.stdout: 262168 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 1 10:52 /home/ubuntu/cephtest , 2020-10-01T10:52:58.724 INFO:teuthology.orchestra.run.smithi035.stdout: 262173 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Oct 1 10:52 /home/ubuntu/cephtest/archive , 2020-10-01T10:52:58.725 INFO:teuthology.orchestra.run.smithi035.stdout: 530790 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 1 10:52 /home/ubuntu/cephtest/archive/audit 2020-10-01T11:24:37.409 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-5799-g5c3ba95b (5c3ba95b4d1208868db1d00d1ba06ed086e1b268) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f8f330b54f8] , 2: () 2020-10-01T11:24:37.412 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:47:45.176 INFO:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.1 config set osd_debug_deep_scrub_sleep 0 , 2020-10-01T10:47:45.319 INFO:teuthology.orchestra.run.smithi121.stdout:{ , 2020-10-01T10:47:45.319 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-10-01T11:24:37.413 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:47:48.757 INFO:tasks.ceph:pgid 10.16 last_scrub_stamp 2020-10-01T10:47:19.514837+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=19, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:47:48.757 INFO:tasks.ceph:pgid 9.15 last_scrub_stamp 2020-10-01T10:47:12.471296+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=12, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:47:48.758 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2020-10-01T10:47:02.501999+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=2, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.415 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:47:48.782 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2020-10-01T10:47:04.511349+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=4, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:47:48.782 INFO:tasks.ceph:pgid 7.9 last_scrub_stamp 2020-10-01T10:47:06.525170+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=6, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:47:48.782 INFO:tasks.ceph:pgid 8.6 last_scrub_stamp 2020-10-01T10:47:09.459489+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=9, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.415 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:48:09.150 INFO:tasks.ceph:pgid 8.12 last_scrub_stamp 2020-10-01T10:47:09.459489+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=9, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:09.151 INFO:tasks.ceph:pgid 10.13 last_scrub_stamp 2020-10-01T10:47:19.514837+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=19, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:09.151 INFO:tasks.ceph:pgid 9.10 last_scrub_stamp 2020-10-01T10:47:12.471296+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=12, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.417 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:48:09.181 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2020-10-01T10:47:04.511349+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=4, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:09.181 INFO:tasks.ceph:pgid 7.f last_scrub_stamp 2020-10-01T10:47:06.525170+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=6, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:09.182 INFO:tasks.ceph:pgid 8.0 last_scrub_stamp 2020-10-01T10:47:09.459489+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=9, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.419 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:48:29.550 INFO:tasks.ceph:pgid 7.14 last_scrub_stamp 2020-10-01T10:47:06.525170+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=6, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:29.550 INFO:tasks.ceph:pgid 10.f last_scrub_stamp 2020-10-01T10:47:19.514837+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=19, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:29.550 INFO:tasks.ceph:pgid 9.c last_scrub_stamp 2020-10-01T10:47:12.471296+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=12, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.421 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:48:49.936 INFO:tasks.ceph:pgid 8.16 last_scrub_stamp 2020-10-01T10:47:09.459489+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=9, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:49.936 INFO:tasks.ceph:pgid 10.17 last_scrub_stamp 2020-10-01T10:47:19.514837+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=19, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:49.937 INFO:tasks.ceph:pgid 9.14 last_scrub_stamp 2020-10-01T10:47:12.471296+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=12, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.436 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:49:10.305 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-01T10:49:30.306 INFO:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-10-01T10:49:30.619 INFO:teuthology.orchestra.run.smithi121.stdout: 2020-10-01T11:24:37.436 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:52:14.638 INFO:tasks.ceph.mon.a:Stopped , 2020-10-01T10:52:14.638 DEBUG:tasks.ceph.mon.c:waiting for process to exit , 2020-10-01T10:52:14.639 INFO:teuthology.orchestra.run:waiting for 300 2020-10-01T11:24:37.437 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:52:27.001 INFO:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , bash: ntpq: command not found , 2020-10-01T10:52:27.020 INFO:teuthology.orchestra.run.smithi162.stdout:210 Number of sources = 4 2020-10-01T11:24:37.437 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:52:58.724 INFO:teuthology.orchestra.run.smithi035.stdout: 262168 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 1 10:52 /home/ubuntu/cephtest , 2020-10-01T10:52:58.724 INFO:teuthology.orchestra.run.smithi035.stdout: 262173 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Oct 1 10:52 /home/ubuntu/cephtest/archive , 2020-10-01T10:52:58.725 INFO:teuthology.orchestra.run.smithi035.stdout: 530790 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 1 10:52 /home/ubuntu/cephtest/archive/audit 2020-10-01T11:24:37.467 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-5799-g5c3ba95b (5c3ba95b4d1208868db1d00d1ba06ed086e1b268) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f8f330b54f8] , 2: () 2020-10-01T11:24:37.470 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:47:45.176 INFO:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.1 config set osd_debug_deep_scrub_sleep 0 , 2020-10-01T10:47:45.319 INFO:teuthology.orchestra.run.smithi121.stdout:{ , 2020-10-01T10:47:45.319 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-10-01T11:24:37.470 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:47:48.757 INFO:tasks.ceph:pgid 10.16 last_scrub_stamp 2020-10-01T10:47:19.514837+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=19, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:47:48.757 INFO:tasks.ceph:pgid 9.15 last_scrub_stamp 2020-10-01T10:47:12.471296+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=12, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:47:48.758 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2020-10-01T10:47:02.501999+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=2, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.472 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:47:48.782 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2020-10-01T10:47:04.511349+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=4, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:47:48.782 INFO:tasks.ceph:pgid 7.9 last_scrub_stamp 2020-10-01T10:47:06.525170+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=6, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:47:48.782 INFO:tasks.ceph:pgid 8.6 last_scrub_stamp 2020-10-01T10:47:09.459489+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=9, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.473 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:48:09.150 INFO:tasks.ceph:pgid 8.12 last_scrub_stamp 2020-10-01T10:47:09.459489+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=9, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:09.151 INFO:tasks.ceph:pgid 10.13 last_scrub_stamp 2020-10-01T10:47:19.514837+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=19, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:09.151 INFO:tasks.ceph:pgid 9.10 last_scrub_stamp 2020-10-01T10:47:12.471296+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=12, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.475 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:48:09.181 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2020-10-01T10:47:04.511349+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=4, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:09.181 INFO:tasks.ceph:pgid 7.f last_scrub_stamp 2020-10-01T10:47:06.525170+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=6, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:09.182 INFO:tasks.ceph:pgid 8.0 last_scrub_stamp 2020-10-01T10:47:09.459489+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=9, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.477 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:48:29.550 INFO:tasks.ceph:pgid 7.14 last_scrub_stamp 2020-10-01T10:47:06.525170+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=6, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:29.550 INFO:tasks.ceph:pgid 10.f last_scrub_stamp 2020-10-01T10:47:19.514837+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=19, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:29.550 INFO:tasks.ceph:pgid 9.c last_scrub_stamp 2020-10-01T10:47:12.471296+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=12, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.478 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:48:49.936 INFO:tasks.ceph:pgid 8.16 last_scrub_stamp 2020-10-01T10:47:09.459489+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=9, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:49.936 INFO:tasks.ceph:pgid 10.17 last_scrub_stamp 2020-10-01T10:47:19.514837+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=19, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:48:49.937 INFO:tasks.ceph:pgid 9.14 last_scrub_stamp 2020-10-01T10:47:12.471296+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=12, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=47, tm_sec=43, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.493 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:49:10.305 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-01T10:49:30.306 INFO:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-10-01T10:49:30.619 INFO:teuthology.orchestra.run.smithi121.stdout: 2020-10-01T11:24:37.494 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:52:14.638 INFO:tasks.ceph.mon.a:Stopped , 2020-10-01T10:52:14.638 DEBUG:tasks.ceph.mon.c:waiting for process to exit , 2020-10-01T10:52:14.639 INFO:teuthology.orchestra.run:waiting for 300 2020-10-01T11:24:37.494 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:52:27.001 INFO:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , bash: ntpq: command not found , 2020-10-01T10:52:27.020 INFO:teuthology.orchestra.run.smithi162.stdout:210 Number of sources = 4 2020-10-01T11:24:37.495 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:52:58.724 INFO:teuthology.orchestra.run.smithi035.stdout: 262168 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 1 10:52 /home/ubuntu/cephtest , 2020-10-01T10:52:58.724 INFO:teuthology.orchestra.run.smithi035.stdout: 262173 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Oct 1 10:52 /home/ubuntu/cephtest/archive , 2020-10-01T10:52:58.725 INFO:teuthology.orchestra.run.smithi035.stdout: 530790 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 1 10:52 /home/ubuntu/cephtest/archive/audit 2020-10-01T11:24:37.549 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-5799-g5c3ba95b (5c3ba95b4d1208868db1d00d1ba06ed086e1b268) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fe3ba7da4f8] , 2: () 2020-10-01T11:24:37.551 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:25:55.107 INFO:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.1 config set osd_debug_deep_scrub_sleep 0 , 2020-10-01T10:25:55.258 INFO:teuthology.orchestra.run.smithi035.stdout:{ , 2020-10-01T10:25:55.258 INFO:teuthology.orchestra.run.smithi035.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-10-01T11:24:37.552 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:25:58.858 INFO:tasks.ceph:pgid 10.16 last_scrub_stamp 2020-10-01T10:25:29.315254+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=29, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=53, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:25:58.858 INFO:tasks.ceph:pgid 9.15 last_scrub_stamp 2020-10-01T10:25:22.278994+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=22, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=53, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:25:58.858 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2020-10-01T10:25:12.287419+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=12, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=53, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.554 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:25:58.883 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2020-10-01T10:25:14.300171+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=14, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=53, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:25:58.883 INFO:tasks.ceph:pgid 7.9 last_scrub_stamp 2020-10-01T10:25:16.315556+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=16, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=53, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:25:58.883 INFO:tasks.ceph:pgid 8.6 last_scrub_stamp 2020-10-01T10:25:19.186597+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=19, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=53, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.555 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:26:19.269 INFO:tasks.ceph:pgid 7.1d last_scrub_stamp 2020-10-01T10:25:16.315556+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=16, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=53, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:26:19.270 INFO:tasks.ceph:pgid 8.12 last_scrub_stamp 2020-10-01T10:25:19.186597+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=19, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=53, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:26:19.270 INFO:tasks.ceph:pgid 10.13 last_scrub_stamp 2020-10-01T10:25:29.315254+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=29, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=53, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.560 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:26:19.296 INFO:tasks.ceph:pgid 9.7 last_scrub_stamp 2020-10-01T10:25:22.278994+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=22, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=53, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:26:19.297 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2020-10-01T10:25:14.300171+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=14, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=53, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:26:19.297 INFO:tasks.ceph:pgid 7.9 last_scrub_stamp 2020-10-01T10:25:16.315556+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=16, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=53, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.561 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:26:39.687 INFO:tasks.ceph:pgid 6.1b last_scrub_stamp 2020-10-01T10:25:14.300171+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=14, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=53, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:26:39.687 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2020-10-01T10:25:16.315556+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=16, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=53, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:26:39.687 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2020-10-01T10:25:19.186597+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=19, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=53, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.564 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:26:59.715 INFO:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-10-01T10:27:00.054 INFO:teuthology.orchestra.run.smithi035.stdout: , dumped all 2020-10-01T11:24:37.628 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:27:20.473 INFO:tasks.ceph:pgid 10.c last_scrub_stamp 2020-10-01T10:25:29.315254+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=29, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=53, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:27:20.473 INFO:tasks.ceph:pgid 9.f last_scrub_stamp 2020-10-01T10:25:22.278994+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=22, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=53, tm_wday=3, tm_yday=275, tm_isdst=0) , 2020-10-01T10:27:20.474 INFO:tasks.ceph:pgid 10.6 last_scrub_stamp 2020-10-01T10:25:29.315254+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=29, tm_wday=3, tm_yday=275, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=1, tm_hour=10, tm_min=25, tm_sec=53, tm_wday=3, tm_yday=275, tm_isdst=0) 2020-10-01T11:24:37.628 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:30:24.865 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-01T10:30:24.929 INFO:tasks.ceph.osd.7:Stopped , 2020-10-01T10:30:24.929 INFO:teuthology.misc:Shutting down mgr daemons... 2020-10-01T11:24:37.629 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-01T10:30:37.255 INFO:teuthology.orchestra.run.smithi035.stdout:^+ hv02.front.sepia.ceph.com 2 6 377 40 +399us[ +398us] +/- 59ms , 2020-10-01T10:30:37.255 INFO:teuthology.orchestra.run.smithi035.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 40 -2708us[-2708us] +/- 73ms , 2020-10-01T10:30:37.256 INFO:teuthology.orchestra.run.smithi035.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 40 -33us[ -33us] +/- 69ms 2020-10-01T11:24:37.629 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 182, in _raise_for_status 2020-10-01T11:24:37.653 INFO:scrape:Found 1 distinct failure reasons 2020-10-01T11:24:37.654 INFO:scrape:Failure: Command failed (workunit test rbd/test_librbd.sh) on smithi162 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c3ba95b4d1208868db1d00d1ba06ed086e1b268 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 RBD_FEATURES=125 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh' 2020-10-01T11:24:37.655 INFO:scrape:2 jobs: ['5485830', '5485824'] 2020-10-01T11:24:37.655 INFO:scrape:suites intersection: ['clusters/{fixed-3', 'min-compat-client/default', 'msgr-failures/few', 'openstack}', 'pool/replicated-data-pool', 'rbd/librbd/{cache/writeback', 'workloads/c_api_tests_with_journaling}'] 2020-10-01T11:24:37.655 INFO:scrape:suites union: ['clusters/{fixed-3', 'config/copy-on-read', 'config/none', 'min-compat-client/default', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-stupid', 'openstack}', 'pool/replicated-data-pool', 'rbd/librbd/{cache/writeback', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'workloads/c_api_tests_with_journaling}'] 2020-10-01T11:24:37.656 INFO:scrape: 2020-10-01T11:24:37.657 INFO:teuthology.results:Sending results to to.my.trociny@gmail.com: Test Run: trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 18 Fail ================================================================= [5485824] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-bitmap pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/c_api_tests_with_journaling} ----------------------------------------------------------------- time: 00:11:33 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485824/ log: http://qa-proxy.ceph.com/teuthology/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485824/ sentry: https://sentry.ceph.com/sentry/sepia/?query=2ffc16538a8f4f24971817df04fede00 Command failed (workunit test rbd/test_librbd.sh) on smithi162 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c3ba95b4d1208868db1d00d1ba06ed086e1b268 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 RBD_FEATURES=125 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh' [5485830] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/none min-compat-client/default msgr-failures/few objectstore/bluestore-stupid pool/replicated-data-pool supported-random-distro$/{rhel_8} workloads/c_api_tests_with_journaling} ----------------------------------------------------------------- time: 00:15:36 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485830/ log: http://qa-proxy.ceph.com/teuthology/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485830/ sentry: https://sentry.ceph.com/sentry/sepia/?query=199611a7d0d14b5796c9dbf317989cf4 Command failed (workunit test rbd/test_librbd.sh) on smithi162 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c3ba95b4d1208868db1d00d1ba06ed086e1b268 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 RBD_FEATURES=125 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh' Pass ================================================================= [5485814] rbd/immutable-object-cache/{clusters/{fix-2 openstack} pool/ceph_and_immutable_object_cache supported-random-distro$/{rhel_8} workloads/c_api_tests_with_defaults} time: 00:24:31 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485814/ [5485815] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-bitmap pool/ec-data-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests} time: 00:22:34 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485815/ [5485816] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-bitmap validator/memcheck workloads/c_api_tests} time: 00:44:57 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485816/ [5485817] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-lz4 pool/none supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_defaults} time: 00:23:53 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485817/ [5485818] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/bluestore-comp-snappy pool/replicated-data-pool supported-random-distro$/{rhel_8} workloads/c_api_tests_with_journaling} time: 01:00:47 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485818/ [5485819] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/filestore-xfs pool/ec-data-pool supported-random-distro$/{centos_8} workloads/c_api_tests} time: 00:20:20 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485819/ [5485820] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-bitmap pool/none supported-random-distro$/{rhel_8} workloads/c_api_tests_with_defaults} time: 00:26:19 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485820/ [5485821] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/none min-compat-client/default msgr-failures/few objectstore/bluestore-comp-lz4 pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/c_api_tests_with_journaling} time: 00:36:43 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485821/ [5485822] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/none min-compat-client/default msgr-failures/few objectstore/bluestore-stupid pool/ec-data-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests} time: 00:22:54 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485822/ [5485823] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/filestore-xfs pool/none supported-random-distro$/{centos_8} workloads/c_api_tests_with_defaults} time: 00:21:27 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485823/ [5485825] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/ec-data-pool supported-random-distro$/{rhel_8} workloads/c_api_tests} time: 00:25:29 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485825/ [5485826] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-stupid pool/none supported-random-distro$/{centos_8} workloads/c_api_tests_with_defaults} time: 00:21:36 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485826/ [5485827] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/filestore-xfs pool/replicated-data-pool supported-random-distro$/{rhel_8} workloads/c_api_tests_with_journaling} time: 00:28:33 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485827/ [5485828] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/bluestore-hybrid pool/ec-data-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests} time: 00:22:18 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485828/ [5485829] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/none supported-random-distro$/{rhel_8} workloads/c_api_tests_with_defaults} time: 00:26:19 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485829/ [5485831] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/none min-compat-client/default msgr-failures/few objectstore/bluestore-comp-zstd pool/ec-data-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests} time: 00:23:31 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485831/ [5485832] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/bluestore-hybrid pool/none supported-random-distro$/{centos_8} workloads/c_api_tests_with_defaults} time: 00:21:53 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485832/ [5485833] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_journaling} time: 00:33:32 info: http://pulpito.ceph.com/trociny-2020-10-01_06:56:16-rbd-wip-mgolub-testing-distro-basic-smithi/5485833/