Found 446 jobs
Ignoring malparsed backtrace:  ceph version 16.0.0-8715-gb26f02a0 (b26f02a08a9d301ae58c79b47e904826b49c4a2c) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7fcc0263fa1b]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x59c) [0x5632e2c15c4c]

Ignoring malparsed backtrace: 2021-01-06T07:29:15.643 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight
, 2021-01-06T07:29:15.758 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument
, 2021-01-06T07:29:15.758 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.

Ignoring malparsed backtrace: 2021-01-06T07:29:37.927 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1
, 2021-01-06T07:29:38.234 INFO:teuthology.orchestra.run.smithi053.stdout:55834574889
, 2021-01-06T07:29:38.249 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574889 got 55834574889 for osd.1

Ignoring malparsed backtrace: 2021-01-06T07:29:48.372 INFO:teuthology.orchestra.run.smithi053.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "
, 2021-01-06T07:29:48.372 INFO:teuthology.orchestra.run.smithi053.stdout:}
, 2021-01-06T07:29:48.390 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7

Ignoring malparsed backtrace: 2021-01-06T07:30:30.365 INFO:teuthology.orchestra.run:waiting for 300
, 2021-01-06T07:30:30.433 INFO:tasks.ceph.osd.3:Stopped
, 2021-01-06T07:30:30.434 DEBUG:tasks.ceph.osd.4:waiting for process to exit

Ignoring malparsed backtrace:   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
,     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2021-01-05-2058/qa/tasks/rados.py", line 277, in task

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 173, in run_tasks
,     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__

Ignoring malparsed backtrace:  ceph version 16.0.0-8715-gb26f02a0 (b26f02a08a9d301ae58c79b47e904826b49c4a2c) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7fcc0263fa1b]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x59c) [0x5632e2c15c4c]

Ignoring malparsed backtrace: 2021-01-06T07:29:15.643 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight
, 2021-01-06T07:29:15.758 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument
, 2021-01-06T07:29:15.758 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.

Ignoring malparsed backtrace: 2021-01-06T07:29:37.927 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1
, 2021-01-06T07:29:38.234 INFO:teuthology.orchestra.run.smithi053.stdout:55834574889
, 2021-01-06T07:29:38.249 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574889 got 55834574889 for osd.1

Ignoring malparsed backtrace: 2021-01-06T07:29:48.372 INFO:teuthology.orchestra.run.smithi053.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "
, 2021-01-06T07:29:48.372 INFO:teuthology.orchestra.run.smithi053.stdout:}
, 2021-01-06T07:29:48.390 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7

Ignoring malparsed backtrace: 2021-01-06T07:30:30.365 INFO:teuthology.orchestra.run:waiting for 300
, 2021-01-06T07:30:30.433 INFO:tasks.ceph.osd.3:Stopped
, 2021-01-06T07:30:30.434 DEBUG:tasks.ceph.osd.4:waiting for process to exit

Ignoring malparsed backtrace:   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
,     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2021-01-05-2058/qa/tasks/rados.py", line 277, in task

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 173, in run_tasks
,     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__

Ignoring malparsed backtrace:  ceph version 16.0.0-8715-gb26f02a0 (b26f02a08a9d301ae58c79b47e904826b49c4a2c) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7fcc0263fa1b]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x59c) [0x5632e2c15c4c]

Ignoring malparsed backtrace: 2021-01-06T07:29:15.643 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight
, 2021-01-06T07:29:15.758 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument
, 2021-01-06T07:29:15.758 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.

Ignoring malparsed backtrace: 2021-01-06T07:29:37.927 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1
, 2021-01-06T07:29:38.234 INFO:teuthology.orchestra.run.smithi053.stdout:55834574889
, 2021-01-06T07:29:38.249 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574889 got 55834574889 for osd.1

Ignoring malparsed backtrace: 2021-01-06T07:29:48.372 INFO:teuthology.orchestra.run.smithi053.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "
, 2021-01-06T07:29:48.372 INFO:teuthology.orchestra.run.smithi053.stdout:}
, 2021-01-06T07:29:48.390 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7

Ignoring malparsed backtrace: 2021-01-06T07:30:30.365 INFO:teuthology.orchestra.run:waiting for 300
, 2021-01-06T07:30:30.433 INFO:tasks.ceph.osd.3:Stopped
, 2021-01-06T07:30:30.434 DEBUG:tasks.ceph.osd.4:waiting for process to exit

Ignoring malparsed backtrace:   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
,     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2021-01-05-2058/qa/tasks/rados.py", line 277, in task

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 173, in run_tasks
,     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__

Ignoring malparsed backtrace:  ceph version 16.0.0-8715-gb26f02a0 (b26f02a08a9d301ae58c79b47e904826b49c4a2c) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7fcc0263fa1b]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x59c) [0x5632e2c15c4c]

Ignoring malparsed backtrace: 2021-01-06T07:29:15.643 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight
, 2021-01-06T07:29:15.758 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument
, 2021-01-06T07:29:15.758 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.

Ignoring malparsed backtrace: 2021-01-06T07:29:37.927 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1
, 2021-01-06T07:29:38.234 INFO:teuthology.orchestra.run.smithi053.stdout:55834574889
, 2021-01-06T07:29:38.249 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574889 got 55834574889 for osd.1

Ignoring malparsed backtrace: 2021-01-06T07:29:48.372 INFO:teuthology.orchestra.run.smithi053.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "
, 2021-01-06T07:29:48.372 INFO:teuthology.orchestra.run.smithi053.stdout:}
, 2021-01-06T07:29:48.390 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7

Ignoring malparsed backtrace: 2021-01-06T07:30:30.365 INFO:teuthology.orchestra.run:waiting for 300
, 2021-01-06T07:30:30.433 INFO:tasks.ceph.osd.3:Stopped
, 2021-01-06T07:30:30.434 DEBUG:tasks.ceph.osd.4:waiting for process to exit

Ignoring malparsed backtrace:   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
,     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2021-01-05-2058/qa/tasks/rados.py", line 277, in task

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 173, in run_tasks
,     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__

Ignoring malparsed backtrace:  ceph version 16.0.0-8715-gb26f02a0 (b26f02a08a9d301ae58c79b47e904826b49c4a2c) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7fce6ab35a1b]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x59c) [0x55bbf7350c4c]

Ignoring malparsed backtrace: 2021-01-06T03:24:09.217 INFO:teuthology.orchestra.run.smithi170.stdout:                        "time": "2021-01-06T03:23:36.987127+0000",
, 2021-01-06T03:24:09.217 INFO:teuthology.orchestra.run.smithi170.stdout:                        "duration": 1.2489e-05
, 2021-01-06T03:24:09.218 INFO:teuthology.orchestra.run.smithi170.stdout:                    },

Ignoring malparsed backtrace: 2021-01-06T03:24:09.254 INFO:teuthology.orchestra.run.smithi170.stdout:            "duration": 1.003554144,
, 2021-01-06T03:24:09.254 INFO:teuthology.orchestra.run.smithi170.stdout:            "type_data": {
, 2021-01-06T03:24:09.255 INFO:teuthology.orchestra.run.smithi170.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-01-06T03:24:09.296 INFO:teuthology.orchestra.run.smithi170.stdout:                        "event": "queued_for_pg",
, 2021-01-06T03:24:09.296 INFO:teuthology.orchestra.run.smithi170.stdout:                        "time": "2021-01-06T03:23:36.989551+0000",
, 2021-01-06T03:24:09.297 INFO:teuthology.orchestra.run.smithi170.stdout:                        "duration": 0.074177409999999999

Ignoring malparsed backtrace: 2021-01-06T03:24:09.322 INFO:teuthology.orchestra.run.smithi170.stdout:                    "client": "client.4586",
, 2021-01-06T03:24:09.322 INFO:teuthology.orchestra.run.smithi170.stdout:                    "client_addr": "v1:172.21.15.170:0/464985562",
, 2021-01-06T03:24:09.323 INFO:teuthology.orchestra.run.smithi170.stdout:                    "tid": 10606

Ignoring malparsed backtrace: 2021-01-06T03:24:09.344 INFO:teuthology.orchestra.run.smithi170.stdout:                    {
, 2021-01-06T03:24:09.345 INFO:teuthology.orchestra.run.smithi170.stdout:                        "event": "reached_pg",
, 2021-01-06T03:24:09.345 INFO:teuthology.orchestra.run.smithi170.stdout:                        "time": "2021-01-06T03:23:37.063851+0000",

Ignoring malparsed backtrace: 2021-01-06T03:24:09.365 INFO:teuthology.orchestra.run.smithi170.stdout:                "events": [
, 2021-01-06T03:24:09.366 INFO:teuthology.orchestra.run.smithi170.stdout:                    {
, 2021-01-06T03:24:09.366 INFO:teuthology.orchestra.run.smithi170.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-01-06T03:24:09.395 INFO:teuthology.orchestra.run.smithi170.stdout:                    },
, 2021-01-06T03:24:09.395 INFO:teuthology.orchestra.run.smithi170.stdout:                    {
, 2021-01-06T03:24:09.396 INFO:teuthology.orchestra.run.smithi170.stdout:                        "event": "waiting for peered",

Ignoring malparsed backtrace: 2021-01-06T03:24:09.422 INFO:teuthology.orchestra.run.smithi170.stdout:                        "duration": 0
, 2021-01-06T03:24:09.422 INFO:teuthology.orchestra.run.smithi170.stdout:                    },
, 2021-01-06T03:24:09.423 INFO:teuthology.orchestra.run.smithi170.stdout:                    {

Ignoring malparsed backtrace: 2021-01-06T03:24:09.443 INFO:teuthology.orchestra.run.smithi170.stdout:                        "duration": 0.92957371700000002
, 2021-01-06T03:24:09.444 INFO:teuthology.orchestra.run.smithi170.stdout:                    },
, 2021-01-06T03:24:09.444 INFO:teuthology.orchestra.run.smithi170.stdout:                    {

Ignoring malparsed backtrace: 2021-01-06T03:24:09.464 INFO:teuthology.orchestra.run.smithi170.stdout:                        "time": "2021-01-06T03:23:37.034009+0000",
, 2021-01-06T03:24:09.465 INFO:teuthology.orchestra.run.smithi170.stdout:                        "duration": 4294967295.9999604
, 2021-01-06T03:24:09.465 INFO:teuthology.orchestra.run.smithi170.stdout:                    },

Ignoring malparsed backtrace: 2021-01-06T03:24:09.485 INFO:teuthology.orchestra.run.smithi170.stdout:                        "time": "2021-01-06T03:23:37.993749+0000",
, 2021-01-06T03:24:09.486 INFO:teuthology.orchestra.run.smithi170.stdout:                        "duration": 1.5065000000000001e-05
, 2021-01-06T03:24:09.486 INFO:teuthology.orchestra.run.smithi170.stdout:                    },

Ignoring malparsed backtrace: 2021-01-06T03:24:09.506 INFO:teuthology.orchestra.run.smithi170.stdout:                        "event": "initiated",
, 2021-01-06T03:24:09.507 INFO:teuthology.orchestra.run.smithi170.stdout:                        "time": "2021-01-06T03:23:37.049767+0000",
, 2021-01-06T03:24:09.507 INFO:teuthology.orchestra.run.smithi170.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-01-06T03:24:09.527 INFO:teuthology.orchestra.run.smithi170.stdout:                        "event": "dispatched",
, 2021-01-06T03:24:09.527 INFO:teuthology.orchestra.run.smithi170.stdout:                        "time": "2021-01-06T03:23:37.050105+0000",
, 2021-01-06T03:24:09.528 INFO:teuthology.orchestra.run.smithi170.stdout:                        "duration": 5.4450000000000004e-06

Ignoring malparsed backtrace: osd.4: osd_enable_op_tracker = 'true'
, osd.5: osd_enable_op_tracker = 'true'
, osd.6: osd_enable_op_tracker = 'true'

Ignoring malparsed backtrace: 2021-01-06T03:24:45.083 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1
, set osd.5 primary-affinity to 1 (8655362)
, 2021-01-06T03:24:46.090 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1

Ignoring malparsed backtrace: 2021-01-06T03:25:00.459 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1
, instructed osd(s) 1 to deep-scrub
, 2021-01-06T03:25:00.784 INFO:tasks.ceph:Scrubbing osd.2

Ignoring malparsed backtrace: 2021-01-06T03:25:24.663 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2021-01-06T03:24:26.287965+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=6, tm_hour=3, tm_min=24, tm_sec=26, tm_wday=2, tm_yday=6, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=6, tm_hour=3, tm_min=24, tm_sec=58, tm_wday=2, tm_yday=6, tm_isdst=0)
, 2021-01-06T03:25:24.663 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2021-01-06T03:24:36.712612+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=6, tm_hour=3, tm_min=24, tm_sec=36, tm_wday=2, tm_yday=6, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=6, tm_hour=3, tm_min=24, tm_sec=58, tm_wday=2, tm_yday=6, tm_isdst=0)
, 2021-01-06T03:25:24.664 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2021-01-06T03:24:37.918102+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=6, tm_hour=3, tm_min=24, tm_sec=37, tm_wday=2, tm_yday=6, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=6, tm_hour=3, tm_min=24, tm_sec=58, tm_wday=2, tm_yday=6, tm_isdst=0)

Ignoring malparsed backtrace: 2021-01-06T03:26:46.587 DEBUG:tasks.ceph.osd.2:waiting for process to exit
, 2021-01-06T03:26:46.587 INFO:teuthology.orchestra.run:waiting for 300
, 2021-01-06T03:26:46.662 INFO:tasks.ceph.osd.2:Stopped

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested
,     yield vars
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 619, in task

Ignoring malparsed backtrace:     raise CommandCrashedError(command=self.command)
, teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base'
, 

Ignoring malparsed backtrace:  ceph version 16.0.0-8715-gb26f02a0 (b26f02a08a9d301ae58c79b47e904826b49c4a2c) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7fce6ab35a1b]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x59c) [0x55bbf7350c4c]

Ignoring malparsed backtrace: 2021-01-06T03:24:09.217 INFO:teuthology.orchestra.run.smithi170.stdout:                        "time": "2021-01-06T03:23:36.987127+0000",
, 2021-01-06T03:24:09.217 INFO:teuthology.orchestra.run.smithi170.stdout:                        "duration": 1.2489e-05
, 2021-01-06T03:24:09.218 INFO:teuthology.orchestra.run.smithi170.stdout:                    },

Ignoring malparsed backtrace: 2021-01-06T03:24:09.254 INFO:teuthology.orchestra.run.smithi170.stdout:            "duration": 1.003554144,
, 2021-01-06T03:24:09.254 INFO:teuthology.orchestra.run.smithi170.stdout:            "type_data": {
, 2021-01-06T03:24:09.255 INFO:teuthology.orchestra.run.smithi170.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-01-06T03:24:09.296 INFO:teuthology.orchestra.run.smithi170.stdout:                        "event": "queued_for_pg",
, 2021-01-06T03:24:09.296 INFO:teuthology.orchestra.run.smithi170.stdout:                        "time": "2021-01-06T03:23:36.989551+0000",
, 2021-01-06T03:24:09.297 INFO:teuthology.orchestra.run.smithi170.stdout:                        "duration": 0.074177409999999999

Ignoring malparsed backtrace: 2021-01-06T03:24:09.322 INFO:teuthology.orchestra.run.smithi170.stdout:                    "client": "client.4586",
, 2021-01-06T03:24:09.322 INFO:teuthology.orchestra.run.smithi170.stdout:                    "client_addr": "v1:172.21.15.170:0/464985562",
, 2021-01-06T03:24:09.323 INFO:teuthology.orchestra.run.smithi170.stdout:                    "tid": 10606

Ignoring malparsed backtrace: 2021-01-06T03:24:09.344 INFO:teuthology.orchestra.run.smithi170.stdout:                    {
, 2021-01-06T03:24:09.345 INFO:teuthology.orchestra.run.smithi170.stdout:                        "event": "reached_pg",
, 2021-01-06T03:24:09.345 INFO:teuthology.orchestra.run.smithi170.stdout:                        "time": "2021-01-06T03:23:37.063851+0000",

Ignoring malparsed backtrace: 2021-01-06T03:24:09.365 INFO:teuthology.orchestra.run.smithi170.stdout:                "events": [
, 2021-01-06T03:24:09.366 INFO:teuthology.orchestra.run.smithi170.stdout:                    {
, 2021-01-06T03:24:09.366 INFO:teuthology.orchestra.run.smithi170.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-01-06T03:24:09.395 INFO:teuthology.orchestra.run.smithi170.stdout:                    },
, 2021-01-06T03:24:09.395 INFO:teuthology.orchestra.run.smithi170.stdout:                    {
, 2021-01-06T03:24:09.396 INFO:teuthology.orchestra.run.smithi170.stdout:                        "event": "waiting for peered",

Ignoring malparsed backtrace: 2021-01-06T03:24:09.422 INFO:teuthology.orchestra.run.smithi170.stdout:                        "duration": 0
, 2021-01-06T03:24:09.422 INFO:teuthology.orchestra.run.smithi170.stdout:                    },
, 2021-01-06T03:24:09.423 INFO:teuthology.orchestra.run.smithi170.stdout:                    {

Ignoring malparsed backtrace: 2021-01-06T03:24:09.443 INFO:teuthology.orchestra.run.smithi170.stdout:                        "duration": 0.92957371700000002
, 2021-01-06T03:24:09.444 INFO:teuthology.orchestra.run.smithi170.stdout:                    },
, 2021-01-06T03:24:09.444 INFO:teuthology.orchestra.run.smithi170.stdout:                    {

Ignoring malparsed backtrace: 2021-01-06T03:24:09.464 INFO:teuthology.orchestra.run.smithi170.stdout:                        "time": "2021-01-06T03:23:37.034009+0000",
, 2021-01-06T03:24:09.465 INFO:teuthology.orchestra.run.smithi170.stdout:                        "duration": 4294967295.9999604
, 2021-01-06T03:24:09.465 INFO:teuthology.orchestra.run.smithi170.stdout:                    },

Ignoring malparsed backtrace: 2021-01-06T03:24:09.485 INFO:teuthology.orchestra.run.smithi170.stdout:                        "time": "2021-01-06T03:23:37.993749+0000",
, 2021-01-06T03:24:09.486 INFO:teuthology.orchestra.run.smithi170.stdout:                        "duration": 1.5065000000000001e-05
, 2021-01-06T03:24:09.486 INFO:teuthology.orchestra.run.smithi170.stdout:                    },

Ignoring malparsed backtrace: 2021-01-06T03:24:09.506 INFO:teuthology.orchestra.run.smithi170.stdout:                        "event": "initiated",
, 2021-01-06T03:24:09.507 INFO:teuthology.orchestra.run.smithi170.stdout:                        "time": "2021-01-06T03:23:37.049767+0000",
, 2021-01-06T03:24:09.507 INFO:teuthology.orchestra.run.smithi170.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-01-06T03:24:09.527 INFO:teuthology.orchestra.run.smithi170.stdout:                        "event": "dispatched",
, 2021-01-06T03:24:09.527 INFO:teuthology.orchestra.run.smithi170.stdout:                        "time": "2021-01-06T03:23:37.050105+0000",
, 2021-01-06T03:24:09.528 INFO:teuthology.orchestra.run.smithi170.stdout:                        "duration": 5.4450000000000004e-06

Ignoring malparsed backtrace: osd.4: osd_enable_op_tracker = 'true'
, osd.5: osd_enable_op_tracker = 'true'
, osd.6: osd_enable_op_tracker = 'true'

Ignoring malparsed backtrace: 2021-01-06T03:24:45.083 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1
, set osd.5 primary-affinity to 1 (8655362)
, 2021-01-06T03:24:46.090 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1

Ignoring malparsed backtrace: 2021-01-06T03:25:00.459 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1
, instructed osd(s) 1 to deep-scrub
, 2021-01-06T03:25:00.784 INFO:tasks.ceph:Scrubbing osd.2

Ignoring malparsed backtrace: 2021-01-06T03:25:24.663 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2021-01-06T03:24:26.287965+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=6, tm_hour=3, tm_min=24, tm_sec=26, tm_wday=2, tm_yday=6, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=6, tm_hour=3, tm_min=24, tm_sec=58, tm_wday=2, tm_yday=6, tm_isdst=0)
, 2021-01-06T03:25:24.663 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2021-01-06T03:24:36.712612+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=6, tm_hour=3, tm_min=24, tm_sec=36, tm_wday=2, tm_yday=6, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=6, tm_hour=3, tm_min=24, tm_sec=58, tm_wday=2, tm_yday=6, tm_isdst=0)
, 2021-01-06T03:25:24.664 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2021-01-06T03:24:37.918102+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=6, tm_hour=3, tm_min=24, tm_sec=37, tm_wday=2, tm_yday=6, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=6, tm_hour=3, tm_min=24, tm_sec=58, tm_wday=2, tm_yday=6, tm_isdst=0)

Ignoring malparsed backtrace: 2021-01-06T03:26:46.587 DEBUG:tasks.ceph.osd.2:waiting for process to exit
, 2021-01-06T03:26:46.587 INFO:teuthology.orchestra.run:waiting for 300
, 2021-01-06T03:26:46.662 INFO:tasks.ceph.osd.2:Stopped

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested
,     yield vars
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 619, in task

Ignoring malparsed backtrace:     raise CommandCrashedError(command=self.command)
, teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base'
, 

Ignoring malparsed backtrace:  ceph version 16.0.0-8715-gb26f02a0 (b26f02a08a9d301ae58c79b47e904826b49c4a2c) pacific (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7ff63351b36d]
,  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7ff63351b548]

Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub
, 2021-01-06T03:43:53.448 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2021-01-06T03:43:53.818 INFO:teuthology.orchestra.run.smithi047.stdout:

Ignoring malparsed backtrace:     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

Ignoring malparsed backtrace:     raise value.with_traceback(tb)
,   File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run
,   File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2021-01-05-2058/qa/tasks/rados.py", line 265, in thread

Ignoring malparsed backtrace:     fetch_binaries_for_coredumps(path, rem)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 318, in fetch_binaries_for_coredumps
,     dump_program = dump_out.split("from '")[1].split(' ')[0]

Ignoring malparsed backtrace:  ceph version 16.0.0-8715-gb26f02a0 (b26f02a08a9d301ae58c79b47e904826b49c4a2c) pacific (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7ff63351b36d]
,  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7ff63351b548]

Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub
, 2021-01-06T03:43:53.448 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2021-01-06T03:43:53.818 INFO:teuthology.orchestra.run.smithi047.stdout:

Ignoring malparsed backtrace:     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

Ignoring malparsed backtrace:     raise value.with_traceback(tb)
,   File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run
,   File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2021-01-05-2058/qa/tasks/rados.py", line 265, in thread

Ignoring malparsed backtrace:     fetch_binaries_for_coredumps(path, rem)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 318, in fetch_binaries_for_coredumps
,     dump_program = dump_out.split("from '")[1].split(' ')[0]

Found 11 distinct failure reasons
Failure: reached maximum tries (180) after waiting for 180 seconds
5 jobs: ['5758281', '5758449', '5758557', '5758228', '5758180']
suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start']
suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{centos_8.0}', 'distro$/{centos_latest}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start']

Dead: b'2021-01-06T20:36:08.345 DEBUG:teuthology.exit:Finished running handlers'
17 jobs: ['5758586', '5758415', '5758312', '5758309', '5758538', '5758261', '5758616', '5758561', '5758465', '5758438', '5758332', '5758613', '5758213', '5758360', '5758411', '5758490', '5758510']
suites intersection: []
suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'backoff/peering', 'ceph', 'cluster/{12-osds', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/objectstore/{backends/objectstore', 'rados/singleton/{all/thrash-eio', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{default}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_striper}', 'thrashers/default', 'thrashers/mapgap', 'thrashers/minsize_recovery', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/cache-snaps}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/ec-small-objects-balanced}', 'workloads/pool-snaps-few-objects}', 'workloads/rados_api_tests}', 'workloads/radosbench-high-concurrency}', 'workloads/redirect}', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}', 'workloads/write_fadvise_dontneed}']

Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base'
4 jobs: ['5758474', '5758182', '5758203', '5758565']
suites intersection: ['rados']
suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/default', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-agent-big}', 'workloads/cache-agent-small}']

Failure: psutil.NoSuchProcess process no longer exists (pid=23344)
1 jobs: ['5758457']
suites: ['ceph', 'clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/ec-rados-plugin=isa-k=2-m=1}']

Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi071 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b26f02a08a9d301ae58c79b47e904826b49c4a2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh'
2 jobs: ['5758389', '5758611']
suites intersection: ['rados/cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}']
suites union: ['rados/cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}']

Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi103 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b26f02a08a9d301ae58c79b47e904826b49c4a2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh'
2 jobs: ['5758454', '5758288']
suites intersection: ['rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}']
suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}']

Timeout 6h running clone.client.0/qa/workunits/rados/test.sh
1 jobs: ['5758543']
suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}']

Failure: Command failed (workunit test scrub/osd-scrub-snaps.sh) on smithi047 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b26f02a08a9d301ae58c79b47e904826b49c4a2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh'
2 jobs: ['5758569', '5758346']
suites intersection: ['supported-random-distro$/{centos_8}', 'workloads/scrub}']
suites union: ['rados/standalone/{mon_election/classic', 'rados/standalone/{mon_election/connectivity', 'supported-random-distro$/{centos_8}', 'workloads/scrub}']

Timeout 3h running clone.client.0/qa/standalone/misc/rados-striper.sh
2 jobs: ['5758475', '5758253']
suites intersection: ['workloads/misc}']
suites union: ['rados/standalone/{mon_election/classic', 'rados/standalone/{mon_election/connectivity', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/misc}']

Timeout 120 running ceph --cluster ceph osd dump --format=json
1 jobs: ['5758501']
suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/pool-snaps-few-objects}']

Failure: Command failed (workunit test rados/test.sh) on smithi101 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b26f02a08a9d301ae58c79b47e904826b49c4a2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
2 jobs: ['5758408', '5758216']
suites intersection: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/classic', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'thrashosds-health', 'workloads/rados_api_tests}']
suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/rados_api_tests}']