2020-12-26T07:25:57.861 INFO:root:teuthology version: 1.0.0-8e49dda4 2020-12-26T07:25:57.862 INFO:teuthology.results:subset: '27/100000' 2020-12-26T07:25:57.862 INFO:teuthology.results:seed: '2698' 2020-12-26T10:55:10.420 INFO:root:teuthology version: 1.0.0-8e49dda4 2020-12-26T10:55:10.481 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-12-26T22:17:38.045 INFO:teuthology.results:Tests finished! gathering results... 2020-12-26T22:18:16.560 INFO:scrape:Found 367 jobs 2020-12-26T22:18:17.630 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f509eb1da2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55da28bcad60] 2020-12-26T22:18:17.632 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-26T08:48:50.951 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-26T08:48:51.247 INFO:teuthology.orchestra.run.smithi013.stdout: 2020-12-26T22:18:17.633 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T08:50:13.350 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T08:50:16.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s , 2020-12-26T08:50:19.452 INFO:tasks.ceph.osd.5:Stopped 2020-12-26T22:18:17.633 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T08:52:10.622 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-26T08:52:10.634 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-26T08:52:10.635 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2020-12-26T22:18:17.739 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f509eb1da2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55da28bcad60] 2020-12-26T22:18:17.741 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-26T08:48:50.951 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-26T08:48:51.247 INFO:teuthology.orchestra.run.smithi013.stdout: 2020-12-26T22:18:17.742 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T08:50:13.350 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T08:50:16.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s , 2020-12-26T08:50:19.452 INFO:tasks.ceph.osd.5:Stopped 2020-12-26T22:18:17.743 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T08:52:10.622 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-26T08:52:10.634 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-26T08:52:10.635 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2020-12-26T22:18:17.846 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f509eb1da2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55da28bcad60] 2020-12-26T22:18:17.848 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-26T08:48:50.951 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-26T08:48:51.247 INFO:teuthology.orchestra.run.smithi013.stdout: 2020-12-26T22:18:17.849 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T08:50:13.350 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T08:50:16.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s , 2020-12-26T08:50:19.452 INFO:tasks.ceph.osd.5:Stopped 2020-12-26T22:18:17.849 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T08:52:10.622 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-26T08:52:10.634 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-26T08:52:10.635 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2020-12-26T22:18:17.946 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f509eb1da2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55da28bcad60] 2020-12-26T22:18:17.948 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-26T08:48:50.951 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-26T08:48:51.247 INFO:teuthology.orchestra.run.smithi013.stdout: 2020-12-26T22:18:17.948 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T08:50:13.350 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T08:50:16.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s , 2020-12-26T08:50:19.452 INFO:tasks.ceph.osd.5:Stopped 2020-12-26T22:18:17.949 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T08:52:10.622 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-26T08:52:10.634 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-26T08:52:10.635 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2020-12-26T22:18:18.047 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f509eb1da2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55da28bcad60] 2020-12-26T22:18:18.049 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-26T08:48:50.951 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-26T08:48:51.247 INFO:teuthology.orchestra.run.smithi013.stdout: 2020-12-26T22:18:18.049 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T08:50:13.350 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T08:50:16.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s , 2020-12-26T08:50:19.452 INFO:tasks.ceph.osd.5:Stopped 2020-12-26T22:18:18.050 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T08:52:10.622 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-26T08:52:10.634 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-26T08:52:10.635 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2020-12-26T22:18:18.167 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f509eb1da2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55da28bcad60] 2020-12-26T22:18:18.169 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-26T08:48:50.951 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-26T08:48:51.247 INFO:teuthology.orchestra.run.smithi013.stdout: 2020-12-26T22:18:18.170 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T08:50:13.350 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T08:50:16.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s , 2020-12-26T08:50:19.452 INFO:tasks.ceph.osd.5:Stopped 2020-12-26T22:18:18.171 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T08:52:10.622 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-26T08:52:10.634 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-26T08:52:10.635 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2020-12-26T22:18:21.003 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f22fb00e826] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55c40baeffe6] 2020-12-26T22:18:21.004 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4381e-05 , 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, , 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: { 2020-12-26T22:18:21.005 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2020-12-26T11:07:59.781651+0000", , 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6019e-05 , 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2020-12-26T22:18:21.006 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:08:10.862+0000 7fbd63052700 -1 osd.7 68 handle_osd_map injecting map crc failure , 2020-12-26T11:08:10.987+0000 7f03bbbc8700 -1 mon.b@0(leader).osd e69 definitely_dead 0 , osd.2 is already down. 2020-12-26T22:18:21.007 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "pglog_hardlimit", , 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "purged_snapdirs", , 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_deletes", 2020-12-26T22:18:21.007 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_count": 0, , 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_target": 0, , 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_barrier": 0, 2020-12-26T22:18:21.008 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "removed_snaps": "[]", , 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_bytes": 0, , 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_objects": 0, 2020-12-26T22:18:21.008 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "min_write_recency_for_promote": 0, , 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_grade_decay_rate": 0, , 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_search_last_n": 0, 2020-12-26T22:18:21.009 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.315 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { , 2020-12-26T11:09:11.316 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ , 2020-12-26T11:09:11.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2020-12-26T22:18:21.010 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.337 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 13, , 2020-12-26T11:09:11.337 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 37, , 2020-12-26T11:09:11.338 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 46, 2020-12-26T22:18:21.010 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.359 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.41:6803/30289", , 2020-12-26T11:09:11.359 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.41:6807/30289", , 2020-12-26T11:09:11.360 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.41:6805/30289", 2020-12-26T22:18:21.011 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.381 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.41:6811", , 2020-12-26T11:09:11.381 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 30319 , 2020-12-26T11:09:11.382 INFO:teuthology.orchestra.run.smithi097.stdout: } 2020-12-26T22:18:21.011 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 , 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, , 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2020-12-26T22:18:21.012 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.425 INFO:teuthology.orchestra.run.smithi097.stdout: } , 2020-12-26T11:09:11.425 INFO:teuthology.orchestra.run.smithi097.stdout:} , 2020-12-26T11:09:11.427 INFO:tasks.thrashosds.thrasher:j is {'epoch': 79, 'fsid': '46f57bee-5f7e-429d-81f8-d0943118d042', 'created': '2020-12-26T11:06:11.553513+0000', 'modified': '2020-12-26T11:09:08.655412+0000', 'last_up_change': '2020-12-26T11:09:07.645903+0000', 'last_in_change': '2020-12-26T11:08:11.729887+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-12-26T11:06:15.200377+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-12-26T11:06:22.371765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 18, 'pg_placement_num': 18, 'pg_placement_num_target': 18, 'pg_num_target': 18, 'pg_num_pending': 18, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'base', 'create_time': '2020-12-26T11:06:35.626198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 3, 'size': 4, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '28', 'last_force_op_resend_prenautilus': '28', 'last_force_op_resend_preluminous': '28', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 26, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [4], 'tier_of': -1, 'read_tier': 4, 'write_tier': 4, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'myprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 4, 'pool_name': 'cache', 'create_time': '2020-12-26T11:06:38.963837+0000', 'flags': 9, 'flags_names': 'hashpspool,incomplete_clones', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '28', 'last_force_op_resend_prenautilus': '28', 'last_force_op_resend_preluminous': '28', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 26, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 3, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 5000, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '30bc335d-671a-49f8-ad9a-b8e11ea38d51', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6800', 'nonce': 30311}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6801', 'nonce': 30311}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6803', 'nonce': 30311}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6802', 'nonce': 30311}]}, 'public_addr': '172.21.15.97:6800/30311', 'cluster_addr': '172.21.15.97:6801/30311', 'heartbeat_back_addr': '172.21.15.97:6803/30311', 'heartbeat_front_addr': '172.21.15.97:6802/30311', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2728a643-5eb6-4f92-ba3e-9145f0784846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6808', 'nonce': 30327}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6809', 'nonce': 30327}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6811', 'nonce': 30327}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6810', 'nonce': 30327}]}, 'public_addr': '172.21.15.97:6808/30327', 'cluster_addr': '172.21.15.97:6809/30327', 'heartbeat_back_addr': '172.21.15.97:6811/30327', 'heartbeat_front_addr': '172.21.15.97:6810/30327', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1d8ad31b-031c-450b-8c81-fb313df9ecd7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 68, 'up_from': 78, 'up_thru': 46, 'down_at': 69, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6812', 'nonce': 52718}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6813', 'nonce': 52718}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6815', 'nonce': 52718}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6814', 'nonce': 52718}]}, 'public_addr': '172.21.15.97:6812/52718', 'cluster_addr': '172.21.15.97:6813/52718', 'heartbeat_back_addr': '172.21.15.97:6815/52718', 'heartbeat_front_addr': '172.21.15.97:6814/52718', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '6c54189f-2275-477b-bcc0-5e45e66510bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 37, 'up_from': 46, 'up_thru': 70, 'down_at': 38, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6804', 'nonce': 43460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6805', 'nonce': 43460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6807', 'nonce': 43460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6806', 'nonce': 43460}]}, 'public_addr': '172.21.15.97:6804/43460', 'cluster_addr': '172.21.15.97:6805/43460', 'heartbeat_back_addr': '172.21.15.97:6807/43460', 'heartbeat_front_addr': '172.21.15.97:6806/43460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'fe41f56e-bc2e-44a3-bb66-6fce32511467', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6801', 'nonce': 30289}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6803', 'nonce': 30289}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6807', 'nonce': 30289}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6805', 'nonce': 30289}]}, 'public_addr': '172.21.15.41:6801/30289', 'cluster_addr': '172.21.15.41:6803/30289', 'heartbeat_back_addr': '172.21.15.41:6807/30289', 'heartbeat_front_addr': '172.21.15.41:6805/30289', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1d05d8b9-8cb2-444c-a590-7a941fa1e189', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6800', 'nonce': 30290}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6802', 'nonce': 30290}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6806', 'nonce': 30290}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6804', 'nonce': 30290}]}, 'public_addr': '172.21.15.41:6800/30290', 'cluster_addr': '172.21.15.41:6802/30290', 'heartbeat_back_addr': '172.21.15.41:6806/30290', 'heartbeat_front_addr': '172.21.15.41:6804/30290', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b0a431da-859f-40ab-a41d-82216ca6c218', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6808', 'nonce': 30319}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6809', 'nonce': 30319}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6811', 'nonce': 30319}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6810', 'nonce': 30319}]}, 'public_addr': '172.21.15.41:6808/30319', 'cluster_addr': '172.21.15.41:6809/30319', 'heartbeat_back_addr': '172.21.15.41:6811/30319', 'heartbeat_front_addr': '172.21.15.41:6810/30319', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ce634247-eafd-40d6-9ffb-afb7893206f4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6812', 'nonce': 30320}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6813', 'nonce': 30320}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6815', 'nonce': 30320}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6814', 'nonce': 30320}]}, 'public_addr': '172.21.15.41:6812/30320', 'cluster_addr': '172.21.15.41:6813/30320', 'heartbeat_back_addr': '172.21.15.41:6815/30320', 'heartbeat_front_addr': '172.21.15.41:6814/30320', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.314520+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.867141+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 65536, 'last_purged_snaps_scrub': '2020-12-26T11:09:08.648413+0000', 'dead_epoch': 69}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:20.277598+0000', 'dead_epoch': 38}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:17.969211+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.121401+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 65536, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.174114+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.323352+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'myprofile': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '2', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2020-12-26T22:18:21.014 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:37.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats , 2020-12-26T11:09:37.691 INFO:teuthology.orchestra.run.smithi097.stdout:55834574890 , 2020-12-26T11:09:37.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2020-12-26T22:18:21.015 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:46.888 INFO:teuthology.orchestra.run.smithi097.stdout:{ , 2020-12-26T11:09:46.889 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2020-12-26T11:09:46.889 INFO:teuthology.orchestra.run.smithi097.stdout:} 2020-12-26T22:18:21.016 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:10:29.194 DEBUG:tasks.ceph.osd.1:waiting for process to exit , 2020-12-26T11:10:29.195 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T11:10:29.261 INFO:tasks.ceph.osd.1:Stopped 2020-12-26T22:18:21.017 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested , yield vars 2020-12-26T22:18:21.017 WARNING:scrape:Ignoring malparsed backtrace: running.get() , File "src/gevent/greenlet.py", line 709, in gevent._greenlet.Greenlet.get , File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception 2020-12-26T22:18:21.108 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f22fb00e826] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55c40baeffe6] 2020-12-26T22:18:21.109 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4381e-05 , 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, , 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: { 2020-12-26T22:18:21.110 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2020-12-26T11:07:59.781651+0000", , 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6019e-05 , 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2020-12-26T22:18:21.111 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:08:10.862+0000 7fbd63052700 -1 osd.7 68 handle_osd_map injecting map crc failure , 2020-12-26T11:08:10.987+0000 7f03bbbc8700 -1 mon.b@0(leader).osd e69 definitely_dead 0 , osd.2 is already down. 2020-12-26T22:18:21.112 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "pglog_hardlimit", , 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "purged_snapdirs", , 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_deletes", 2020-12-26T22:18:21.112 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_count": 0, , 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_target": 0, , 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_barrier": 0, 2020-12-26T22:18:21.113 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "removed_snaps": "[]", , 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_bytes": 0, , 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_objects": 0, 2020-12-26T22:18:21.113 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "min_write_recency_for_promote": 0, , 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_grade_decay_rate": 0, , 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_search_last_n": 0, 2020-12-26T22:18:21.114 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.315 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { , 2020-12-26T11:09:11.316 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ , 2020-12-26T11:09:11.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2020-12-26T22:18:21.114 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.337 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 13, , 2020-12-26T11:09:11.337 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 37, , 2020-12-26T11:09:11.338 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 46, 2020-12-26T22:18:21.115 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.359 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.41:6803/30289", , 2020-12-26T11:09:11.359 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.41:6807/30289", , 2020-12-26T11:09:11.360 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.41:6805/30289", 2020-12-26T22:18:21.116 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.381 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.41:6811", , 2020-12-26T11:09:11.381 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 30319 , 2020-12-26T11:09:11.382 INFO:teuthology.orchestra.run.smithi097.stdout: } 2020-12-26T22:18:21.116 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 , 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, , 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2020-12-26T22:18:21.117 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.425 INFO:teuthology.orchestra.run.smithi097.stdout: } , 2020-12-26T11:09:11.425 INFO:teuthology.orchestra.run.smithi097.stdout:} , 2020-12-26T11:09:11.427 INFO:tasks.thrashosds.thrasher:j is {'epoch': 79, 'fsid': '46f57bee-5f7e-429d-81f8-d0943118d042', 'created': '2020-12-26T11:06:11.553513+0000', 'modified': '2020-12-26T11:09:08.655412+0000', 'last_up_change': '2020-12-26T11:09:07.645903+0000', 'last_in_change': '2020-12-26T11:08:11.729887+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-12-26T11:06:15.200377+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-12-26T11:06:22.371765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 18, 'pg_placement_num': 18, 'pg_placement_num_target': 18, 'pg_num_target': 18, 'pg_num_pending': 18, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'base', 'create_time': '2020-12-26T11:06:35.626198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 3, 'size': 4, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '28', 'last_force_op_resend_prenautilus': '28', 'last_force_op_resend_preluminous': '28', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 26, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [4], 'tier_of': -1, 'read_tier': 4, 'write_tier': 4, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'myprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 4, 'pool_name': 'cache', 'create_time': '2020-12-26T11:06:38.963837+0000', 'flags': 9, 'flags_names': 'hashpspool,incomplete_clones', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '28', 'last_force_op_resend_prenautilus': '28', 'last_force_op_resend_preluminous': '28', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 26, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 3, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 5000, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '30bc335d-671a-49f8-ad9a-b8e11ea38d51', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6800', 'nonce': 30311}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6801', 'nonce': 30311}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6803', 'nonce': 30311}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6802', 'nonce': 30311}]}, 'public_addr': '172.21.15.97:6800/30311', 'cluster_addr': '172.21.15.97:6801/30311', 'heartbeat_back_addr': '172.21.15.97:6803/30311', 'heartbeat_front_addr': '172.21.15.97:6802/30311', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2728a643-5eb6-4f92-ba3e-9145f0784846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6808', 'nonce': 30327}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6809', 'nonce': 30327}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6811', 'nonce': 30327}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6810', 'nonce': 30327}]}, 'public_addr': '172.21.15.97:6808/30327', 'cluster_addr': '172.21.15.97:6809/30327', 'heartbeat_back_addr': '172.21.15.97:6811/30327', 'heartbeat_front_addr': '172.21.15.97:6810/30327', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1d8ad31b-031c-450b-8c81-fb313df9ecd7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 68, 'up_from': 78, 'up_thru': 46, 'down_at': 69, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6812', 'nonce': 52718}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6813', 'nonce': 52718}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6815', 'nonce': 52718}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6814', 'nonce': 52718}]}, 'public_addr': '172.21.15.97:6812/52718', 'cluster_addr': '172.21.15.97:6813/52718', 'heartbeat_back_addr': '172.21.15.97:6815/52718', 'heartbeat_front_addr': '172.21.15.97:6814/52718', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '6c54189f-2275-477b-bcc0-5e45e66510bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 37, 'up_from': 46, 'up_thru': 70, 'down_at': 38, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6804', 'nonce': 43460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6805', 'nonce': 43460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6807', 'nonce': 43460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6806', 'nonce': 43460}]}, 'public_addr': '172.21.15.97:6804/43460', 'cluster_addr': '172.21.15.97:6805/43460', 'heartbeat_back_addr': '172.21.15.97:6807/43460', 'heartbeat_front_addr': '172.21.15.97:6806/43460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'fe41f56e-bc2e-44a3-bb66-6fce32511467', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6801', 'nonce': 30289}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6803', 'nonce': 30289}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6807', 'nonce': 30289}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6805', 'nonce': 30289}]}, 'public_addr': '172.21.15.41:6801/30289', 'cluster_addr': '172.21.15.41:6803/30289', 'heartbeat_back_addr': '172.21.15.41:6807/30289', 'heartbeat_front_addr': '172.21.15.41:6805/30289', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1d05d8b9-8cb2-444c-a590-7a941fa1e189', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6800', 'nonce': 30290}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6802', 'nonce': 30290}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6806', 'nonce': 30290}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6804', 'nonce': 30290}]}, 'public_addr': '172.21.15.41:6800/30290', 'cluster_addr': '172.21.15.41:6802/30290', 'heartbeat_back_addr': '172.21.15.41:6806/30290', 'heartbeat_front_addr': '172.21.15.41:6804/30290', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b0a431da-859f-40ab-a41d-82216ca6c218', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6808', 'nonce': 30319}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6809', 'nonce': 30319}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6811', 'nonce': 30319}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6810', 'nonce': 30319}]}, 'public_addr': '172.21.15.41:6808/30319', 'cluster_addr': '172.21.15.41:6809/30319', 'heartbeat_back_addr': '172.21.15.41:6811/30319', 'heartbeat_front_addr': '172.21.15.41:6810/30319', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ce634247-eafd-40d6-9ffb-afb7893206f4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6812', 'nonce': 30320}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6813', 'nonce': 30320}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6815', 'nonce': 30320}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6814', 'nonce': 30320}]}, 'public_addr': '172.21.15.41:6812/30320', 'cluster_addr': '172.21.15.41:6813/30320', 'heartbeat_back_addr': '172.21.15.41:6815/30320', 'heartbeat_front_addr': '172.21.15.41:6814/30320', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.314520+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.867141+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 65536, 'last_purged_snaps_scrub': '2020-12-26T11:09:08.648413+0000', 'dead_epoch': 69}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:20.277598+0000', 'dead_epoch': 38}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:17.969211+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.121401+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 65536, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.174114+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.323352+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'myprofile': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '2', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2020-12-26T22:18:21.119 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:37.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats , 2020-12-26T11:09:37.691 INFO:teuthology.orchestra.run.smithi097.stdout:55834574890 , 2020-12-26T11:09:37.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2020-12-26T22:18:21.120 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:46.888 INFO:teuthology.orchestra.run.smithi097.stdout:{ , 2020-12-26T11:09:46.889 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2020-12-26T11:09:46.889 INFO:teuthology.orchestra.run.smithi097.stdout:} 2020-12-26T22:18:21.121 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:10:29.194 DEBUG:tasks.ceph.osd.1:waiting for process to exit , 2020-12-26T11:10:29.195 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T11:10:29.261 INFO:tasks.ceph.osd.1:Stopped 2020-12-26T22:18:21.121 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested , yield vars 2020-12-26T22:18:21.122 WARNING:scrape:Ignoring malparsed backtrace: running.get() , File "src/gevent/greenlet.py", line 709, in gevent._greenlet.Greenlet.get , File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception 2020-12-26T22:18:21.215 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f22fb00e826] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55c40baeffe6] 2020-12-26T22:18:21.216 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4381e-05 , 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, , 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: { 2020-12-26T22:18:21.216 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2020-12-26T11:07:59.781651+0000", , 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6019e-05 , 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2020-12-26T22:18:21.217 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:08:10.862+0000 7fbd63052700 -1 osd.7 68 handle_osd_map injecting map crc failure , 2020-12-26T11:08:10.987+0000 7f03bbbc8700 -1 mon.b@0(leader).osd e69 definitely_dead 0 , osd.2 is already down. 2020-12-26T22:18:21.218 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "pglog_hardlimit", , 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "purged_snapdirs", , 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_deletes", 2020-12-26T22:18:21.219 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_count": 0, , 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_target": 0, , 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_barrier": 0, 2020-12-26T22:18:21.219 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "removed_snaps": "[]", , 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_bytes": 0, , 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_objects": 0, 2020-12-26T22:18:21.220 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "min_write_recency_for_promote": 0, , 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_grade_decay_rate": 0, , 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_search_last_n": 0, 2020-12-26T22:18:21.220 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.315 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { , 2020-12-26T11:09:11.316 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ , 2020-12-26T11:09:11.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2020-12-26T22:18:21.221 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.337 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 13, , 2020-12-26T11:09:11.337 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 37, , 2020-12-26T11:09:11.338 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 46, 2020-12-26T22:18:21.221 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.359 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.41:6803/30289", , 2020-12-26T11:09:11.359 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.41:6807/30289", , 2020-12-26T11:09:11.360 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.41:6805/30289", 2020-12-26T22:18:21.222 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.381 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.41:6811", , 2020-12-26T11:09:11.381 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 30319 , 2020-12-26T11:09:11.382 INFO:teuthology.orchestra.run.smithi097.stdout: } 2020-12-26T22:18:21.223 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 , 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, , 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2020-12-26T22:18:21.224 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.425 INFO:teuthology.orchestra.run.smithi097.stdout: } , 2020-12-26T11:09:11.425 INFO:teuthology.orchestra.run.smithi097.stdout:} , 2020-12-26T11:09:11.427 INFO:tasks.thrashosds.thrasher:j is {'epoch': 79, 'fsid': '46f57bee-5f7e-429d-81f8-d0943118d042', 'created': '2020-12-26T11:06:11.553513+0000', 'modified': '2020-12-26T11:09:08.655412+0000', 'last_up_change': '2020-12-26T11:09:07.645903+0000', 'last_in_change': '2020-12-26T11:08:11.729887+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-12-26T11:06:15.200377+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-12-26T11:06:22.371765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 18, 'pg_placement_num': 18, 'pg_placement_num_target': 18, 'pg_num_target': 18, 'pg_num_pending': 18, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'base', 'create_time': '2020-12-26T11:06:35.626198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 3, 'size': 4, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '28', 'last_force_op_resend_prenautilus': '28', 'last_force_op_resend_preluminous': '28', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 26, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [4], 'tier_of': -1, 'read_tier': 4, 'write_tier': 4, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'myprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 4, 'pool_name': 'cache', 'create_time': '2020-12-26T11:06:38.963837+0000', 'flags': 9, 'flags_names': 'hashpspool,incomplete_clones', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '28', 'last_force_op_resend_prenautilus': '28', 'last_force_op_resend_preluminous': '28', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 26, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 3, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 5000, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '30bc335d-671a-49f8-ad9a-b8e11ea38d51', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6800', 'nonce': 30311}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6801', 'nonce': 30311}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6803', 'nonce': 30311}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6802', 'nonce': 30311}]}, 'public_addr': '172.21.15.97:6800/30311', 'cluster_addr': '172.21.15.97:6801/30311', 'heartbeat_back_addr': '172.21.15.97:6803/30311', 'heartbeat_front_addr': '172.21.15.97:6802/30311', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2728a643-5eb6-4f92-ba3e-9145f0784846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6808', 'nonce': 30327}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6809', 'nonce': 30327}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6811', 'nonce': 30327}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6810', 'nonce': 30327}]}, 'public_addr': '172.21.15.97:6808/30327', 'cluster_addr': '172.21.15.97:6809/30327', 'heartbeat_back_addr': '172.21.15.97:6811/30327', 'heartbeat_front_addr': '172.21.15.97:6810/30327', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1d8ad31b-031c-450b-8c81-fb313df9ecd7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 68, 'up_from': 78, 'up_thru': 46, 'down_at': 69, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6812', 'nonce': 52718}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6813', 'nonce': 52718}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6815', 'nonce': 52718}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6814', 'nonce': 52718}]}, 'public_addr': '172.21.15.97:6812/52718', 'cluster_addr': '172.21.15.97:6813/52718', 'heartbeat_back_addr': '172.21.15.97:6815/52718', 'heartbeat_front_addr': '172.21.15.97:6814/52718', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '6c54189f-2275-477b-bcc0-5e45e66510bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 37, 'up_from': 46, 'up_thru': 70, 'down_at': 38, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6804', 'nonce': 43460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6805', 'nonce': 43460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6807', 'nonce': 43460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6806', 'nonce': 43460}]}, 'public_addr': '172.21.15.97:6804/43460', 'cluster_addr': '172.21.15.97:6805/43460', 'heartbeat_back_addr': '172.21.15.97:6807/43460', 'heartbeat_front_addr': '172.21.15.97:6806/43460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'fe41f56e-bc2e-44a3-bb66-6fce32511467', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6801', 'nonce': 30289}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6803', 'nonce': 30289}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6807', 'nonce': 30289}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6805', 'nonce': 30289}]}, 'public_addr': '172.21.15.41:6801/30289', 'cluster_addr': '172.21.15.41:6803/30289', 'heartbeat_back_addr': '172.21.15.41:6807/30289', 'heartbeat_front_addr': '172.21.15.41:6805/30289', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1d05d8b9-8cb2-444c-a590-7a941fa1e189', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6800', 'nonce': 30290}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6802', 'nonce': 30290}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6806', 'nonce': 30290}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6804', 'nonce': 30290}]}, 'public_addr': '172.21.15.41:6800/30290', 'cluster_addr': '172.21.15.41:6802/30290', 'heartbeat_back_addr': '172.21.15.41:6806/30290', 'heartbeat_front_addr': '172.21.15.41:6804/30290', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b0a431da-859f-40ab-a41d-82216ca6c218', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6808', 'nonce': 30319}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6809', 'nonce': 30319}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6811', 'nonce': 30319}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6810', 'nonce': 30319}]}, 'public_addr': '172.21.15.41:6808/30319', 'cluster_addr': '172.21.15.41:6809/30319', 'heartbeat_back_addr': '172.21.15.41:6811/30319', 'heartbeat_front_addr': '172.21.15.41:6810/30319', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ce634247-eafd-40d6-9ffb-afb7893206f4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6812', 'nonce': 30320}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6813', 'nonce': 30320}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6815', 'nonce': 30320}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6814', 'nonce': 30320}]}, 'public_addr': '172.21.15.41:6812/30320', 'cluster_addr': '172.21.15.41:6813/30320', 'heartbeat_back_addr': '172.21.15.41:6815/30320', 'heartbeat_front_addr': '172.21.15.41:6814/30320', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.314520+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.867141+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 65536, 'last_purged_snaps_scrub': '2020-12-26T11:09:08.648413+0000', 'dead_epoch': 69}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:20.277598+0000', 'dead_epoch': 38}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:17.969211+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.121401+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 65536, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.174114+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.323352+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'myprofile': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '2', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2020-12-26T22:18:21.225 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:37.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats , 2020-12-26T11:09:37.691 INFO:teuthology.orchestra.run.smithi097.stdout:55834574890 , 2020-12-26T11:09:37.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2020-12-26T22:18:21.227 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:46.888 INFO:teuthology.orchestra.run.smithi097.stdout:{ , 2020-12-26T11:09:46.889 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2020-12-26T11:09:46.889 INFO:teuthology.orchestra.run.smithi097.stdout:} 2020-12-26T22:18:21.227 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:10:29.194 DEBUG:tasks.ceph.osd.1:waiting for process to exit , 2020-12-26T11:10:29.195 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T11:10:29.261 INFO:tasks.ceph.osd.1:Stopped 2020-12-26T22:18:21.228 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested , yield vars 2020-12-26T22:18:21.229 WARNING:scrape:Ignoring malparsed backtrace: running.get() , File "src/gevent/greenlet.py", line 709, in gevent._greenlet.Greenlet.get , File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception 2020-12-26T22:18:21.320 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f22fb00e826] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55c40baeffe6] 2020-12-26T22:18:21.321 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4381e-05 , 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, , 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: { 2020-12-26T22:18:21.322 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2020-12-26T11:07:59.781651+0000", , 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6019e-05 , 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2020-12-26T22:18:21.323 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:08:10.862+0000 7fbd63052700 -1 osd.7 68 handle_osd_map injecting map crc failure , 2020-12-26T11:08:10.987+0000 7f03bbbc8700 -1 mon.b@0(leader).osd e69 definitely_dead 0 , osd.2 is already down. 2020-12-26T22:18:21.324 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "pglog_hardlimit", , 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "purged_snapdirs", , 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_deletes", 2020-12-26T22:18:21.324 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_count": 0, , 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_target": 0, , 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_barrier": 0, 2020-12-26T22:18:21.325 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "removed_snaps": "[]", , 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_bytes": 0, , 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_objects": 0, 2020-12-26T22:18:21.325 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "min_write_recency_for_promote": 0, , 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_grade_decay_rate": 0, , 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_search_last_n": 0, 2020-12-26T22:18:21.326 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.315 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { , 2020-12-26T11:09:11.316 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ , 2020-12-26T11:09:11.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2020-12-26T22:18:21.327 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.337 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 13, , 2020-12-26T11:09:11.337 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 37, , 2020-12-26T11:09:11.338 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 46, 2020-12-26T22:18:21.327 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.359 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.41:6803/30289", , 2020-12-26T11:09:11.359 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.41:6807/30289", , 2020-12-26T11:09:11.360 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.41:6805/30289", 2020-12-26T22:18:21.328 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.381 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.41:6811", , 2020-12-26T11:09:11.381 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 30319 , 2020-12-26T11:09:11.382 INFO:teuthology.orchestra.run.smithi097.stdout: } 2020-12-26T22:18:21.328 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 , 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, , 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2020-12-26T22:18:21.329 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:11.425 INFO:teuthology.orchestra.run.smithi097.stdout: } , 2020-12-26T11:09:11.425 INFO:teuthology.orchestra.run.smithi097.stdout:} , 2020-12-26T11:09:11.427 INFO:tasks.thrashosds.thrasher:j is {'epoch': 79, 'fsid': '46f57bee-5f7e-429d-81f8-d0943118d042', 'created': '2020-12-26T11:06:11.553513+0000', 'modified': '2020-12-26T11:09:08.655412+0000', 'last_up_change': '2020-12-26T11:09:07.645903+0000', 'last_in_change': '2020-12-26T11:08:11.729887+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-12-26T11:06:15.200377+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-12-26T11:06:22.371765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 18, 'pg_placement_num': 18, 'pg_placement_num_target': 18, 'pg_num_target': 18, 'pg_num_pending': 18, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'base', 'create_time': '2020-12-26T11:06:35.626198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 3, 'size': 4, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '28', 'last_force_op_resend_prenautilus': '28', 'last_force_op_resend_preluminous': '28', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 26, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [4], 'tier_of': -1, 'read_tier': 4, 'write_tier': 4, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'myprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 4, 'pool_name': 'cache', 'create_time': '2020-12-26T11:06:38.963837+0000', 'flags': 9, 'flags_names': 'hashpspool,incomplete_clones', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '28', 'last_force_op_resend_prenautilus': '28', 'last_force_op_resend_preluminous': '28', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 26, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 3, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 5000, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '30bc335d-671a-49f8-ad9a-b8e11ea38d51', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6800', 'nonce': 30311}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6801', 'nonce': 30311}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6803', 'nonce': 30311}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6802', 'nonce': 30311}]}, 'public_addr': '172.21.15.97:6800/30311', 'cluster_addr': '172.21.15.97:6801/30311', 'heartbeat_back_addr': '172.21.15.97:6803/30311', 'heartbeat_front_addr': '172.21.15.97:6802/30311', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2728a643-5eb6-4f92-ba3e-9145f0784846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6808', 'nonce': 30327}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6809', 'nonce': 30327}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6811', 'nonce': 30327}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6810', 'nonce': 30327}]}, 'public_addr': '172.21.15.97:6808/30327', 'cluster_addr': '172.21.15.97:6809/30327', 'heartbeat_back_addr': '172.21.15.97:6811/30327', 'heartbeat_front_addr': '172.21.15.97:6810/30327', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1d8ad31b-031c-450b-8c81-fb313df9ecd7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 68, 'up_from': 78, 'up_thru': 46, 'down_at': 69, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6812', 'nonce': 52718}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6813', 'nonce': 52718}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6815', 'nonce': 52718}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6814', 'nonce': 52718}]}, 'public_addr': '172.21.15.97:6812/52718', 'cluster_addr': '172.21.15.97:6813/52718', 'heartbeat_back_addr': '172.21.15.97:6815/52718', 'heartbeat_front_addr': '172.21.15.97:6814/52718', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '6c54189f-2275-477b-bcc0-5e45e66510bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 37, 'up_from': 46, 'up_thru': 70, 'down_at': 38, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6804', 'nonce': 43460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6805', 'nonce': 43460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6807', 'nonce': 43460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6806', 'nonce': 43460}]}, 'public_addr': '172.21.15.97:6804/43460', 'cluster_addr': '172.21.15.97:6805/43460', 'heartbeat_back_addr': '172.21.15.97:6807/43460', 'heartbeat_front_addr': '172.21.15.97:6806/43460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'fe41f56e-bc2e-44a3-bb66-6fce32511467', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6801', 'nonce': 30289}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6803', 'nonce': 30289}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6807', 'nonce': 30289}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6805', 'nonce': 30289}]}, 'public_addr': '172.21.15.41:6801/30289', 'cluster_addr': '172.21.15.41:6803/30289', 'heartbeat_back_addr': '172.21.15.41:6807/30289', 'heartbeat_front_addr': '172.21.15.41:6805/30289', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1d05d8b9-8cb2-444c-a590-7a941fa1e189', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6800', 'nonce': 30290}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6802', 'nonce': 30290}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6806', 'nonce': 30290}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6804', 'nonce': 30290}]}, 'public_addr': '172.21.15.41:6800/30290', 'cluster_addr': '172.21.15.41:6802/30290', 'heartbeat_back_addr': '172.21.15.41:6806/30290', 'heartbeat_front_addr': '172.21.15.41:6804/30290', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b0a431da-859f-40ab-a41d-82216ca6c218', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6808', 'nonce': 30319}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6809', 'nonce': 30319}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6811', 'nonce': 30319}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6810', 'nonce': 30319}]}, 'public_addr': '172.21.15.41:6808/30319', 'cluster_addr': '172.21.15.41:6809/30319', 'heartbeat_back_addr': '172.21.15.41:6811/30319', 'heartbeat_front_addr': '172.21.15.41:6810/30319', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ce634247-eafd-40d6-9ffb-afb7893206f4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6812', 'nonce': 30320}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6813', 'nonce': 30320}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6815', 'nonce': 30320}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6814', 'nonce': 30320}]}, 'public_addr': '172.21.15.41:6812/30320', 'cluster_addr': '172.21.15.41:6813/30320', 'heartbeat_back_addr': '172.21.15.41:6815/30320', 'heartbeat_front_addr': '172.21.15.41:6814/30320', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.314520+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.867141+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 65536, 'last_purged_snaps_scrub': '2020-12-26T11:09:08.648413+0000', 'dead_epoch': 69}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:20.277598+0000', 'dead_epoch': 38}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:17.969211+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.121401+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 65536, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.174114+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.323352+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'myprofile': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '2', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2020-12-26T22:18:21.331 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:37.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats , 2020-12-26T11:09:37.691 INFO:teuthology.orchestra.run.smithi097.stdout:55834574890 , 2020-12-26T11:09:37.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2020-12-26T22:18:21.332 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:09:46.888 INFO:teuthology.orchestra.run.smithi097.stdout:{ , 2020-12-26T11:09:46.889 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2020-12-26T11:09:46.889 INFO:teuthology.orchestra.run.smithi097.stdout:} 2020-12-26T22:18:21.333 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-26T11:10:29.194 DEBUG:tasks.ceph.osd.1:waiting for process to exit , 2020-12-26T11:10:29.195 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T11:10:29.261 INFO:tasks.ceph.osd.1:Stopped 2020-12-26T22:18:21.334 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested , yield vars 2020-12-26T22:18:21.334 WARNING:scrape:Ignoring malparsed backtrace: running.get() , File "src/gevent/greenlet.py", line 709, in gevent._greenlet.Greenlet.get , File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception 2020-12-26T22:18:23.580 INFO:scrape:Found 9 distinct failure reasons 2020-12-26T22:18:23.582 INFO:scrape:Failure: reached maximum tries (180) after waiting for 180 seconds 2020-12-26T22:18:23.582 INFO:scrape:4 jobs: ['5737972', '5737878', '5737941', '5738104'] 2020-12-26T22:18:23.583 INFO:scrape:suites intersection: [] 2020-12-26T22:18:23.583 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'distro$/{centos_8.0}', 'distro$/{centos_latest}', 'distro$/{rhel_8.0}', 'fixed-2', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity}', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/cephadm/upgrade/{1-start', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/radosbench-high-concurrency}'] 2020-12-26T22:18:23.584 INFO:scrape: 2020-12-26T22:18:23.585 INFO:scrape:Failure: "2020-12-26T09:57:15.727976+0000 osd.0 (osd.0) 143 : cluster [ERR] scrub 59.1d 59:bbaa2602:test-rados-api-smithi007-85713-74::foo:24 : size 0 != clone_size 10" in cluster log 2020-12-26T22:18:23.585 INFO:scrape:1 jobs: ['5737927'] 2020-12-26T22:18:23.585 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/rados_api_tests}'] 2020-12-26T22:18:23.586 INFO:scrape: 2020-12-26T22:18:23.587 INFO:scrape:Failure: "2020-12-26T10:22:23.521304+0000 mon.a (mon.0) 11 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log 2020-12-26T22:18:23.587 INFO:scrape:1 jobs: ['5738019'] 2020-12-26T22:18:23.588 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'rados', 'rados/singleton/{all/osd-recovery-incomplete', 'supported-random-distro$/{rhel_8}}'] 2020-12-26T22:18:23.588 INFO:scrape: 2020-12-26T22:18:23.589 INFO:scrape:Dead: b'2020-12-26T22:15:25.573 DEBUG:teuthology.exit:Finished running handlers' 2020-12-26T22:18:23.589 INFO:scrape:9 jobs: ['5738030', '5738021', '5738004', '5737963', '5737939', '5737871', '5737757', '5737752', '5737970'] 2020-12-26T22:18:23.590 INFO:scrape:suites intersection: ['openstack}', 'rados', 'thrashosds-health'] 2020-12-26T22:18:23.590 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'cluster/{12-osds', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'fast/fast', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{default}', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-pool-snaps}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/ec-small-objects-many-deletes}', 'workloads/redirect}', 'workloads/small-objects-localized}', 'workloads/small-objects}'] 2020-12-26T22:18:23.591 INFO:scrape: 2020-12-26T22:18:23.591 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2020-12-26T22:18:23.592 INFO:scrape:2 jobs: ['5737826', '5738100'] 2020-12-26T22:18:23.592 INFO:scrape:suites intersection: ['mon_election/classic', 'msgr-failures/few', 'rados'] 2020-12-26T22:18:23.593 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-agent-big}'] 2020-12-26T22:18:23.594 INFO:scrape: 2020-12-26T22:18:23.594 INFO:scrape:Failure: Command failed (workunit test cephtool/test.sh) on smithi138 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=43e363d3957ee0c8e0e948d4d3e58a75abda2778 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2020-12-26T22:18:23.595 INFO:scrape:1 jobs: ['5737767'] 2020-12-26T22:18:23.595 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_8}}'] 2020-12-26T22:18:23.596 INFO:scrape: 2020-12-26T22:18:23.596 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/rados/test.sh 2020-12-26T22:18:23.597 INFO:scrape:2 jobs: ['5737901', '5738044'] 2020-12-26T22:18:23.597 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] 2020-12-26T22:18:23.598 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] 2020-12-26T22:18:23.598 INFO:scrape: 2020-12-26T22:18:23.599 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json 2020-12-26T22:18:23.599 INFO:scrape:1 jobs: ['5737756'] 2020-12-26T22:18:23.600 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/small-objects-localized}'] 2020-12-26T22:18:23.600 INFO:scrape: 2020-12-26T22:18:23.601 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=43e363d3957ee0c8e0e948d4d3e58a75abda2778 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2020-12-26T22:18:23.601 INFO:scrape:2 jobs: ['5737953', '5737819'] 2020-12-26T22:18:23.602 INFO:scrape:suites intersection: ['rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}'] 2020-12-26T22:18:23.602 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}'] 2020-12-26T22:18:23.603 INFO:scrape: 2020-12-26T22:18:23.615 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/ failed: 14 dead: 9 running: 0 waiting: 0 queued: 0 passed: 344 Fail ================================================================= [5737756] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 09:23:54 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737756/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737756/ Command failed on smithi136 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [5737767] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:32:33 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737767/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737767/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6c1212e78f124c09ada359698056b6b7 Command failed (workunit test cephtool/test.sh) on smithi138 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=43e363d3957ee0c8e0e948d4d3e58a75abda2778 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' [5737819] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:14:55 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737819/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737819/ sentry: https://sentry.ceph.com/organizations/ceph/?query=13fe394c4d944db9bf08bde3d55f86f1 Command failed (workunit test cephadm/test_cephadm.sh) on smithi103 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=43e363d3957ee0c8e0e948d4d3e58a75abda2778 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [5737826] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:11:40 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737826/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737826/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [5737878] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:18:44 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737878/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737878/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fca8132fdbce46e89cbbe9d76ee6b331 reached maximum tries (180) after waiting for 180 seconds [5737901] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 06:47:33 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737901/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737901/ sentry: https://sentry.ceph.com/organizations/ceph/?query=10da0e0285524187bd4e4d82c52c419b Command failed (workunit test rados/test.sh) on smithi045 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=43e363d3957ee0c8e0e948d4d3e58a75abda2778 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5737927] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:35:11 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737927/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737927/ "2020-12-26T09:57:15.727976+0000 osd.0 (osd.0) 143 : cluster [ERR] scrub 59.1d 59:bbaa2602:test-rados-api-smithi007-85713-74::foo:24 : size 0 != clone_size 10" in cluster log [5737941] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 03:32:42 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737941/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737941/ sentry: https://sentry.ceph.com/organizations/ceph/?query=002338736d2143e9b2f4e9faa2214e8a reached maximum tries (500) after waiting for 3000 seconds [5737953] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:14:53 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737953/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737953/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f23b46b13f2641aeade363795d881303 Command failed (workunit test cephadm/test_cephadm.sh) on smithi118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=43e363d3957ee0c8e0e948d4d3e58a75abda2778 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [5737972] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{centos_8.0} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:17:58 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737972/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737972/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2fd40bfc2b314f52a0d844c2a6e05f33 reached maximum tries (180) after waiting for 180 seconds [5738019] rados/singleton/{all/osd-recovery-incomplete mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:18:16 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738019/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738019/ "2020-12-26T10:22:23.521304+0000 mon.a (mon.0) 11 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log [5738044] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 06:46:38 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738044/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738044/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5836004b25cb4003b80634e3352542e2 Command failed (workunit test rados/test.sh) on smithi063 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=43e363d3957ee0c8e0e948d4d3e58a75abda2778 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5738100] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:13:18 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738100/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738100/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [5738104] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{centos_latest} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:17:08 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738104/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738104/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d3439ac93c645348f23bb52b99e5692 reached maximum tries (180) after waiting for 180 seconds Dead ================================================================= [5737752] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737752/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737752/ [5737757] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737757/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737757/ [5737871] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737871/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737871/ [5737939] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737939/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737939/ [5737963] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737963/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737963/ [5737970] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737970/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737970/ [5738004] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-many-deletes} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738004/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738004/ [5738021] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738021/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738021/ [5738030] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738030/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738030/ Pass ================================================================= [5737745] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:28:42 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737745/ [5737746] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:06:05 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737746/ [5737747] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:51 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737747/ [5737748] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:34:55 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737748/ [5737749] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:08:38 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737749/ [5737750] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects} time: 00:23:11 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737750/ [5737751] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} tasks/module_selftest} time: 00:28:47 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737751/ [5737753] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/mon_recovery validater/valgrind} time: 00:27:25 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737753/ [5737754] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} time: 00:18:51 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737754/ [5737755] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start} time: 00:13:43 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737755/ [5737758] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:11:45 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737758/ [5737759] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:16:45 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737759/ [5737760] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:27:48 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737760/ [5737761] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_big} time: 00:39:17 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737761/ [5737762] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/many workloads/rados_5925} time: 00:10:27 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737762/ [5737763] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:06:59 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737763/ [5737764] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start} time: 00:12:59 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737764/ [5737765] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/small-objects} time: 00:23:13 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737765/ [5737766] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/test_rbd_api} time: 00:19:46 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737766/ [5737768] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:09:57 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737768/ [5737769] rados/objectstore/{backends/alloc-hint supported-random-distro$/{rhel_8}} time: 00:16:48 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737769/ [5737770] rados/rest/{mgr-restful supported-random-distro$/{centos_8}} time: 00:16:51 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737770/ [5737771] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:15:57 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737771/ [5737772] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/crush} time: 00:18:46 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737772/ [5737773] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/classic thrashosds-health ubuntu_latest} time: 05:17:58 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737773/ [5737774] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:19:23 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737774/ [5737775] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:09:00 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737775/ [5737776] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_adoption} time: 00:16:00 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737776/ [5737777] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:17:06 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737777/ [5737778] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:33:09 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737778/ [5737779] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/snaps-few-objects-balanced} time: 00:24:24 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737779/ [5737780] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{centos_8}} time: 00:13:05 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737780/ [5737781] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:17:13 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737781/ [5737782] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:56 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737782/ [5737783] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:13:46 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737783/ [5737784] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:25:56 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737784/ [5737785] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:20:09 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737785/ [5737786] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/progress} time: 00:16:03 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737786/ [5737787] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:08:14 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737787/ [5737788] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:33:00 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737788/ [5737789] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:12:59 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737789/ [5737790] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:12:50 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737790/ [5737791] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} time: 00:33:04 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737791/ [5737792] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:07:30 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737792/ [5737793] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:27:41 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737793/ [5737794] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{ubuntu_18.04} fixed-2 mon_election/classic} time: 00:18:02 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737794/ [5737795] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:06:34 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737795/ [5737796] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mix} time: 00:26:45 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737796/ [5737797] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:21:51 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737797/ [5737798] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_read_write} time: 00:20:30 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737798/ [5737799] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 01:14:24 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737799/ [5737800] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 00:40:19 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737800/ [5737801] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/erasure-code} time: 00:42:03 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737801/ [5737802] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/cache-snaps} time: 00:30:38 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737802/ [5737803] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_api_tests} time: 00:34:09 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737803/ [5737804] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:25:50 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737804/ [5737805] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:13:30 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737805/ [5737806] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:13:21 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737806/ [5737807] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:43:16 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737807/ [5737808] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start} time: 00:15:14 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737808/ [5737809] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:12:26 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737809/ [5737810] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:09:05 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737810/ [5737811] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}} time: 00:10:17 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737811/ [5737812] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:15:06 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737812/ [5737813] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:11:50 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737813/ [5737814] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/radosbench} time: 00:59:21 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737814/ [5737815] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:27:44 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737815/ [5737816] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_write} time: 00:26:30 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737816/ [5737817] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:41 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737817/ [5737818] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_python} time: 00:29:13 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737818/ [5737820] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/prometheus} time: 00:12:00 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737820/ [5737821] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mostlyread} time: 00:38:38 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737821/ [5737822] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/mgr} time: 00:19:29 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737822/ [5737823] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:11:04 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737823/ [5737824] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:27:36 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737824/ [5737825] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:08:18 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737825/ [5737827] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/admin_socket_objecter_requests} time: 00:17:27 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737827/ [5737828] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:12:56 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737828/ [5737829] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:30:20 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737829/ [5737830] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/rados_cls_all validater/valgrind} time: 02:25:12 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737830/ [5737831] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:50 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737831/ [5737832] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_18.04_podman} fixed-2 mon_election/connectivity} time: 00:19:46 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737832/ [5737833] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:09:57 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737833/ [5737834] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/rbd_cls} time: 00:19:30 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737834/ [5737835] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:09:25 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737835/ [5737836] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:06:11 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737836/ [5737837] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} time: 00:24:30 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737837/ [5737838] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-agent-big} time: 00:34:35 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737838/ [5737839] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/rados_mon_osdmap_prune} time: 00:54:49 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737839/ [5737840] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:12:50 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737840/ [5737841] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:10:27 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737841/ [5737842] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:13:55 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737842/ [5737843] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:40:08 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737843/ [5737844] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:05:37 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737844/ [5737845] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:17:27 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737845/ [5737846] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-agent-small} time: 00:10:09 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737846/ [5737847] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start} time: 00:13:04 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737847/ [5737848] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/misc} time: 00:32:47 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737848/ [5737849] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm_repos} time: 00:13:24 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737849/ [5737850] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:07:32 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737850/ [5737851] rados/objectstore/{backends/filejournal supported-random-distro$/{ubuntu_latest}} time: 00:06:25 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737851/ [5737852] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:44 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737852/ [5737853] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} tasks/readwrite} time: 00:13:16 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737853/ [5737854] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:17:04 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737854/ [5737855] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:37:25 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737855/ [5737856] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:13:36 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737856/ [5737857] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{centos_8} tasks/workunits} time: 00:15:59 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737857/ [5737858] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:32:01 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737858/ [5737859] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start} time: 00:12:52 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737859/ [5737860] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:08:36 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737860/ [5737861] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:27:09 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737861/ [5737862] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:46:36 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737862/ [5737863] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:11:45 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737863/ [5737864] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:12:05 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737864/ [5737865] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/lockdep} time: 00:15:12 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737865/ [5737866] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:11:37 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737866/ [5737867] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:05 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737867/ [5737868] rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:10:33 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737868/ [5737869] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:32:03 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737869/ [5737870] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:06:57 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737870/ [5737872] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/test_rbd_api} time: 00:18:50 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737872/ [5737873] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:58 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737873/ [5737874] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/sync workloads/rados_mon_workunits} time: 00:35:34 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737874/ [5737875] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:14:28 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737875/ [5737876] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/mon} time: 00:50:07 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737876/ [5737877] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:09:43 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737877/ [5737879] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:14:07 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737879/ [5737880] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_orch_cli} time: 00:17:29 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737880/ [5737881] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-snaps-balanced} time: 00:20:33 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737881/ [5737882] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/repair_test} time: 00:14:48 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737882/ [5737883] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:08:48 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737883/ [5737884] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:13:04 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737884/ [5737885] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:24 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737885/ [5737886] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:07:30 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737886/ [5737887] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/few rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:30:33 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737887/ [5737888] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:15:21 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737888/ [5737889] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:11:36 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737889/ [5737890] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:08:33 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737890/ [5737891] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/dashboard} time: 01:09:10 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737891/ [5737892] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{ubuntu_latest}} time: 02:28:25 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737892/ [5737893] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:19:23 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737893/ [5737894] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 00:40:24 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737894/ [5737895] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-snaps} time: 00:22:34 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737895/ [5737896] rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:46:02 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737896/ [5737897] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:08:47 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737897/ [5737898] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-radosbench} time: 02:13:07 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737898/ [5737899] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/cache-snaps} time: 00:30:32 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737899/ [5737900] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:28:20 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737900/ [5737902] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:22:48 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737902/ [5737903] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start} time: 00:15:11 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737903/ [5737904] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/osd} time: 03:11:48 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737904/ [5737905] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:36 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737905/ [5737906] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/radosbench} time: 01:37:26 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737906/ [5737907] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache} time: 00:09:48 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737907/ [5737908] rados/singleton/{all/erasure-code-nonregression mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:14:54 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737908/ [5737909] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:09:11 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737909/ [5737910] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:12:53 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737910/ [5737911] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/force-sync-many workloads/snaps-few-objects} time: 00:30:09 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737911/ [5737912] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:16:06 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737912/ [5737913] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:19:12 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737913/ [5737914] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:49 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737914/ [5737915] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/scrub_test} time: 00:08:58 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737915/ [5737916] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_python} time: 00:28:17 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737916/ [5737917] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/pool-snaps-few-objects} time: 00:21:27 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737917/ [5737918] rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:45:34 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737918/ [5737919] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_adoption} time: 00:15:56 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737919/ [5737920] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:44 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737920/ [5737921] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:16:27 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737921/ [5737922] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{ubuntu_18.04_podman} fixed-2 mon_election/connectivity} time: 00:18:55 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737922/ [5737923] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/failover} time: 00:13:16 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737923/ [5737924] rados/singleton/{all/lost-unfound mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:50:06 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737924/ [5737925] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:12:52 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737925/ [5737926] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/scrub} time: 01:45:26 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737926/ [5737928] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:13:06 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737928/ [5737929] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/rbd_cls} time: 00:15:58 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737929/ [5737930] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:29:34 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737930/ [5737931] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/mon_recovery} time: 00:23:49 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737931/ [5737932] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-balanced} time: 00:21:20 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737932/ [5737933] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:05:22 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737933/ [5737934] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{rhel_8}} time: 02:29:27 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737934/ [5737935] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:08:17 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737935/ [5737936] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados tasks/rados_cls_all validater/lockdep} time: 00:24:04 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737936/ [5737937] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start} time: 00:13:26 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737937/ [5737938] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/connectivity msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:08:25 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737938/ [5737940] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:11:32 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737940/ [5737942] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:22:19 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737942/ [5737943] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} time: 00:39:20 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737943/ [5737944] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:13:55 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737944/ [5737945] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} time: 00:30:28 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737945/ [5737946] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:36:01 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737946/ [5737947] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/many workloads/pool-create-delete} time: 00:18:27 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737947/ [5737948] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:12:43 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737948/ [5737949] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start} time: 00:12:34 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737949/ [5737950] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/radosbench} time: 01:55:06 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737950/ [5737951] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:43:05 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737951/ [5737952] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:11:00 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737952/ [5737954] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:30 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737954/ [5737955] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:16:33 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737955/ [5737956] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/crush} time: 00:18:38 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737956/ [5737957] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:46 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737957/ [5737958] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/connectivity thrashosds-health ubuntu_latest} time: 04:44:03 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737958/ [5737959] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:31:59 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737959/ [5737960] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:12:25 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737960/ [5737961] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{ubuntu_latest}} time: 00:10:22 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737961/ [5737962] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:10:44 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737962/ [5737964] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:17:32 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737964/ [5737965] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:14:13 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737965/ [5737966] rados/singleton/{all/mon-auth-caps mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:16:51 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737966/ [5737967] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:12:56 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737967/ [5737968] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:08:30 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737968/ [5737969] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read} time: 00:23:02 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737969/ [5737971] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_cls_all validater/valgrind} time: 02:23:33 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737971/ [5737973] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/redirect_promote_tests} time: 00:13:57 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737973/ [5737974] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:07:22 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737974/ [5737975] rados/objectstore/{backends/fusestore supported-random-distro$/{ubuntu_latest}} time: 00:12:26 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737975/ [5737976] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:12 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737976/ [5737977] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} tasks/rados_cls_all} time: 00:25:39 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737977/ [5737978] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:06:31 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737978/ [5737979] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:09:39 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737979/ [5737980] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} time: 00:40:45 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737980/ [5737981] rados/singleton/{all/mon-config-key-caps mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} time: 00:18:24 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737981/ [5737982] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:18:25 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737982/ [5737983] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/erasure-code} time: 00:44:35 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737983/ [5737984] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/one workloads/rados_5925} time: 00:09:39 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737984/ [5737985] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:13:30 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737985/ [5737986] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/redirect_set_object} time: 00:11:42 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737986/ [5737987] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:16:53 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737987/ [5737988] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm_repos} time: 00:13:16 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737988/ [5737989] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:10 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737989/ [5737990] rados/singleton/{all/mon-config-keys mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:21:36 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737990/ [5737991] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} time: 00:28:45 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737991/ [5737992] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} time: 00:22:35 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737992/ [5737993] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:07:45 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737993/ [5737994] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:29:26 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737994/ [5737995] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/set-chunks-read} time: 00:15:12 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737995/ [5737996] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/module_selftest} time: 00:28:23 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737996/ [5737997] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:14:49 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737997/ [5737998] rados/singleton/{all/mon-config mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:14:03 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737998/ [5737999] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:09:44 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5737999/ [5738000] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start} time: 00:14:48 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738000/ [5738001] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:12:42 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738001/ [5738002] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/rados_python} time: 00:22:35 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738002/ [5738003] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/filestore-xfs rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:08:01 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738003/ [5738005] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/mgr} time: 00:18:48 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738005/ [5738006] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:12:30 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738006/ [5738007] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:09:41 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738007/ [5738008] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/mon_recovery validater/lockdep} time: 00:12:04 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738008/ [5738009] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects-balanced} time: 00:19:37 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738009/ [5738010] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:05:48 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738010/ [5738011] rados/singleton/{all/osd-backfill mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:20:11 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738011/ [5738012] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} time: 01:28:42 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738012/ [5738013] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:16 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738013/ [5738014] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_18.04} fixed-2 mon_election/connectivity} time: 00:17:15 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738014/ [5738015] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:38 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738015/ [5738016] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}} time: 00:12:16 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738016/ [5738017] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:19:01 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738017/ [5738018] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:42:29 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738018/ [5738020] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:08:57 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738020/ [5738022] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/sync-many workloads/rados_api_tests} time: 00:34:44 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738022/ [5738023] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:13:01 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738023/ [5738024] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:09:31 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738024/ [5738025] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:29 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738025/ [5738026] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:14:08 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738026/ [5738027] rados/singleton/{all/osd-recovery mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:11:05 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738027/ [5738028] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_orch_cli} time: 00:18:00 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738028/ [5738029] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:17:41 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738029/ [5738031] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} tasks/progress} time: 00:15:45 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738031/ [5738032] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/misc} time: 00:32:53 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738032/ [5738033] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start} time: 00:13:51 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738033/ [5738034] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:07:34 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738034/ [5738035] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/rados_stress_watch} time: 00:23:54 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738035/ [5738036] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:19:43 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738036/ [5738037] rados/singleton/{all/peer mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:08:59 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738037/ [5738038] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:16:57 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738038/ [5738039] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:29:38 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738039/ [5738040] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-balanced} time: 00:29:20 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738040/ [5738041] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:13:27 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738041/ [5738042] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-small-objects} time: 00:19:22 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738042/ [5738043] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:30:47 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738043/ [5738045] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start} time: 00:12:57 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738045/ [5738046] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:08:32 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738046/ [5738047] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:45:54 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738047/ [5738048] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:26:00 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738048/ [5738049] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:30:38 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738049/ [5738050] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:12:03 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738050/ [5738051] rados/singleton/{all/pg-autoscaler-progress-off mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:15:01 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738051/ [5738052] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:07:33 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738052/ [5738053] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:37:38 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738053/ [5738054] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-localized} time: 00:22:00 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738054/ [5738055] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_adoption} time: 00:13:48 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738055/ [5738056] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{centos_8} tasks/dashboard} time: 01:10:03 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738056/ [5738057] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/sync workloads/rados_mon_osdmap_prune} time: 00:59:04 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738057/ [5738058] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{ubuntu_latest}} time: 00:34:03 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738058/ [5738059] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:52 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738059/ [5738060] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/mon} time: 00:49:30 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738060/ [5738061] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:13:56 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738061/ [5738062] rados/singleton/{all/pg-autoscaler mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:16:38 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738062/ [5738063] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:17:14 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738063/ [5738064] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_latest} fixed-2 mon_election/classic} time: 00:17:39 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738064/ [5738065] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:25:59 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738065/ [5738066] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} tasks/prometheus} time: 00:12:36 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738066/ [5738067] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 00:38:45 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738067/ [5738068] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/rados_striper} time: 00:10:14 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738068/ [5738069] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:24:08 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738069/ [5738070] rados/singleton/{all/pg-removal-interruption mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} time: 00:08:28 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738070/ [5738071] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:13:07 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738071/ [5738072] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:14:54 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738072/ [5738073] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:11:30 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738073/ [5738074] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/write_fadvise_dontneed} time: 00:16:15 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738074/ [5738075] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/dashboard} time: 01:09:20 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738075/ [5738076] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/test_rbd_api} time: 00:15:04 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738076/ [5738077] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:08:27 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738077/ [5738078] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:30:56 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738078/ [5738079] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:08:44 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738079/ [5738080] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_cls_all validater/lockdep} time: 00:28:55 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738080/ [5738081] rados/singleton/{all/radostool mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:22:18 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738081/ [5738082] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start} time: 00:15:11 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738082/ [5738083] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:38:20 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738083/ [5738084] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:19:24 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738084/ [5738085] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/osd} time: 03:12:16 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738085/ [5738086] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm} time: 00:20:49 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738086/ [5738087] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:51:01 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738087/ [5738088] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:29:57 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738088/ [5738089] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:54 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738089/ [5738090] rados/singleton/{all/random-eio mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:24:12 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738090/ [5738091] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:11:53 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738091/ [5738092] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_mon_workunits} time: 00:36:21 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738092/ [5738093] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} time: 00:28:28 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738093/ [5738094] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:19:16 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738094/ [5738095] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:07:08 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738095/ [5738096] rados/objectstore/{backends/objectstore supported-random-distro$/{centos_8}} time: 05:13:56 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738096/ [5738097] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:12:46 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738097/ [5738098] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:34:23 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738098/ [5738099] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_big} time: 00:37:31 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738099/ [5738101] rados/singleton/{all/rebuild-mondb mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:14:07 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738101/ [5738102] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:18:54 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738102/ [5738103] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{centos_8} tasks/workunits} time: 00:16:18 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738103/ [5738105] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:13:17 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738105/ [5738106] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:12:47 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738106/ [5738107] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-agent-small} time: 00:15:19 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738107/ [5738108] rados/singleton/{all/recovery-preemption mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:18:05 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738108/ [5738109] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/scrub} time: 01:46:10 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738109/ [5738110] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/radosbench} time: 01:13:42 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738110/ [5738111] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:08:51 info: http://pulpito.ceph.com/teuthology-2020-12-26_07:01:01-rados-master-distro-basic-smithi/5738111/