2022-12-03T13:09:23.766 INFO:root:teuthology version: 0.0.1.dev3+g4da97cf 2022-12-03T13:09:23.768 INFO:teuthology.results:seed: '411' 2022-12-03T13:19:55.307 INFO:root:teuthology version: 0.0.1.dev3+g4da97cf 2022-12-03T13:19:55.311 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-12-03T16:34:09.720 INFO:teuthology.results:Tests finished! gathering results... 2022-12-03T16:34:10.713 INFO:scrape:Found 12 jobs 2022-12-03T16:34:11.272 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-1099-g4a79121c (4a79121cfd1ee1750ffed71ede253747b30e6436) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fc24b493c0e] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x564241ce70ca] 2022-12-03T16:34:11.273 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:06:13.338+0000 7fc57c5c3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2022-12-03T14:06:13.346+0000 7fc57c5c3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , reweighted osd.2 to 1 (10000) 2022-12-03T16:34:11.275 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:06:21.327 DEBUG:teuthology.parallel:result is None , 2022-12-03T14:06:21.339 INFO:teuthology.orchestra.run.smithi026.stdout:38654707494 , 2022-12-03T14:06:21.354 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654707493 got 38654707494 for osd.0 2022-12-03T16:34:11.276 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:06:27.444+0000 7fb9a3766700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2022-12-03T14:06:27.450+0000 7fb9a3766700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2022-12-03T14:06:27.648 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-12-03T16:34:11.276 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:07:07.499 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi161.front.sepia.ceph.com , 2022-12-03T14:07:07.500 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 , 2022-12-03T14:07:07.620 INFO:tasks.ceph:Archiving mon data... 2022-12-03T16:34:11.277 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:07:41.125 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi161.front.sepia.ceph.com , 2022-12-03T14:07:41.126 DEBUG:teuthology.orchestra.run.smithi161:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2022-12-03T14:07:41.149 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2022-12-03T16:34:11.401 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-1099-g4a79121c (4a79121cfd1ee1750ffed71ede253747b30e6436) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fc24b493c0e] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x564241ce70ca] 2022-12-03T16:34:11.403 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:06:13.338+0000 7fc57c5c3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2022-12-03T14:06:13.346+0000 7fc57c5c3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , reweighted osd.2 to 1 (10000) 2022-12-03T16:34:11.404 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:06:21.327 DEBUG:teuthology.parallel:result is None , 2022-12-03T14:06:21.339 INFO:teuthology.orchestra.run.smithi026.stdout:38654707494 , 2022-12-03T14:06:21.354 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654707493 got 38654707494 for osd.0 2022-12-03T16:34:11.405 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:06:27.444+0000 7fb9a3766700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2022-12-03T14:06:27.450+0000 7fb9a3766700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2022-12-03T14:06:27.648 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-12-03T16:34:11.406 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:07:07.499 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi161.front.sepia.ceph.com , 2022-12-03T14:07:07.500 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 , 2022-12-03T14:07:07.620 INFO:tasks.ceph:Archiving mon data... 2022-12-03T16:34:11.406 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:07:41.125 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi161.front.sepia.ceph.com , 2022-12-03T14:07:41.126 DEBUG:teuthology.orchestra.run.smithi161:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2022-12-03T14:07:41.149 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2022-12-03T16:34:11.548 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-1099-g4a79121c (4a79121cfd1ee1750ffed71ede253747b30e6436) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fc24b493c0e] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x564241ce70ca] 2022-12-03T16:34:11.550 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:06:13.338+0000 7fc57c5c3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2022-12-03T14:06:13.346+0000 7fc57c5c3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , reweighted osd.2 to 1 (10000) 2022-12-03T16:34:11.551 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:06:21.327 DEBUG:teuthology.parallel:result is None , 2022-12-03T14:06:21.339 INFO:teuthology.orchestra.run.smithi026.stdout:38654707494 , 2022-12-03T14:06:21.354 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654707493 got 38654707494 for osd.0 2022-12-03T16:34:11.552 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:06:27.444+0000 7fb9a3766700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2022-12-03T14:06:27.450+0000 7fb9a3766700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2022-12-03T14:06:27.648 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-12-03T16:34:11.553 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:07:07.499 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi161.front.sepia.ceph.com , 2022-12-03T14:07:07.500 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 , 2022-12-03T14:07:07.620 INFO:tasks.ceph:Archiving mon data... 2022-12-03T16:34:11.553 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:07:41.125 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi161.front.sepia.ceph.com , 2022-12-03T14:07:41.126 DEBUG:teuthology.orchestra.run.smithi161:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2022-12-03T14:07:41.149 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2022-12-03T16:34:11.674 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-1099-g4a79121c (4a79121cfd1ee1750ffed71ede253747b30e6436) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fc24b493c0e] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x564241ce70ca] 2022-12-03T16:34:11.675 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:06:13.338+0000 7fc57c5c3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2022-12-03T14:06:13.346+0000 7fc57c5c3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , reweighted osd.2 to 1 (10000) 2022-12-03T16:34:11.677 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:06:21.327 DEBUG:teuthology.parallel:result is None , 2022-12-03T14:06:21.339 INFO:teuthology.orchestra.run.smithi026.stdout:38654707494 , 2022-12-03T14:06:21.354 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654707493 got 38654707494 for osd.0 2022-12-03T16:34:11.678 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:06:27.444+0000 7fb9a3766700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2022-12-03T14:06:27.450+0000 7fb9a3766700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2022-12-03T14:06:27.648 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-12-03T16:34:11.678 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:07:07.499 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi161.front.sepia.ceph.com , 2022-12-03T14:07:07.500 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 , 2022-12-03T14:07:07.620 INFO:tasks.ceph:Archiving mon data... 2022-12-03T16:34:11.679 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:07:41.125 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi161.front.sepia.ceph.com , 2022-12-03T14:07:41.126 DEBUG:teuthology.orchestra.run.smithi161:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2022-12-03T14:07:41.149 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2022-12-03T16:34:12.206 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-1099-g4a79121c (4a79121cfd1ee1750ffed71ede253747b30e6436) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f62aaca1c0e] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5581b57dd0ca] 2022-12-03T16:34:12.208 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:07:54.288 INFO:teuthology.orchestra.run.smithi033.stdout: , 2022-12-03T14:07:54.289 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":222,"fsid":"c5b2e7cc-1e84-45cd-990c-39e1017fbde1","created":"2022-12-03T13:37:20.046999+0000","modified":"2022-12-03T14:07:53.579217+0000","last_up_change":"2022-12-03T13:37:28.012321+0000","last_in_change":"2022-12-03T13:37:22.349737+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-12-03T13:37:29.765153+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-12-03T13:37:30.676877+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2022-12-03T13:37:37.583768+0000","flags":262305,"flags_names":"hashpspool,nopgchange,write_fadvise_dontneed,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"40ba3a2d-f0c0-44fb-86e7-067e28dae341","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6801","nonce":103950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":103950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":103950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6805","nonce":103950}]},"public_addr":"172.21.15.33:6801/103950","cluster_addr":"172.21.15.33:6800/103950","heartbeat_back_addr":"172.21.15.33:6804/103950","heartbeat_front_addr":"172.21.15.33:6805/103950","state":["exists","up"]},{"osd":1,"uuid":"4159bd15-e111-4941-8aaf-10f5848b26bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":103977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6803","nonce":103977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6807","nonce":103977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":103977}]},"public_addr":"172.21.15.33:6802/103977","cluster_addr":"172.21.15.33:6803/103977","heartbeat_back_addr":"172.21.15.33:6807/103977","heartbeat_front_addr":"172.21.15.33:6806/103977","state":["exists","up"]},{"osd":2,"uuid":"3acfb27e-258d-432f-a122-4bea5518efdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6803","nonce":103548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":103548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6807","nonce":103548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":103548}]},"public_addr":"172.21.15.133:6803/103548","cluster_addr":"172.21.15.133:6802/103548","heartbeat_back_addr":"172.21.15.133:6807/103548","heartbeat_front_addr":"172.21.15.133:6806/103548","state":["exists","up"]},{"osd":3,"uuid":"d14bac50-f443-464f-82a7-b5d0406dcf89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":103573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6805","nonce":103573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6809","nonce":103573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":103573}]},"public_addr":"172.21.15.133:6804/103573","cluster_addr":"172.21.15.133:6805/103573","heartbeat_back_addr":"172.21.15.133:6809/103573","heartbeat_front_addr":"172.21.15.133:6808/103573","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2022-12-03T14:07:54.304 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2022-12-03T16:34:12.209 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:07:58.852 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2022-12-03T14:07:58.979+0000 7fa0b835b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2022-12-03T14:07:58.985+0000 7fa0b835b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2022-12-03T16:34:12.210 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:08:04.003 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2022-12-03T13:37:38.178755+0000 time.struct_time(tm_year=2022, tm_mon=12, tm_mday=3, tm_hour=13, tm_min=37, tm_sec=38, tm_wday=5, tm_yday=337, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=12, tm_mday=3, tm_hour=14, tm_min=8, tm_sec=0, tm_wday=5, tm_yday=337, tm_isdst=0) , 2022-12-03T14:08:04.003 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2022-12-03T13:37:31.022058+0000 time.struct_time(tm_year=2022, tm_mon=12, tm_mday=3, tm_hour=13, tm_min=37, tm_sec=31, tm_wday=5, tm_yday=337, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=12, tm_mday=3, tm_hour=14, tm_min=8, tm_sec=0, tm_wday=5, tm_yday=337, tm_isdst=0) , 2022-12-03T14:08:04.004 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2022-12-03T13:37:31.022058+0000 time.struct_time(tm_year=2022, tm_mon=12, tm_mday=3, tm_hour=13, tm_min=37, tm_sec=31, tm_wday=5, tm_yday=337, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=12, tm_mday=3, tm_hour=14, tm_min=8, tm_sec=0, tm_wday=5, tm_yday=337, tm_isdst=0) 2022-12-03T16:34:12.211 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:08:50.434 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar cz -f - -C /var/lib/ceph/crash -- . , 2022-12-03T14:08:50.473 INFO:tasks.ceph:Compressing logs... , 2022-12-03T14:08:50.474 DEBUG:teuthology.orchestra.run.smithi033:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-12-03T16:34:12.212 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:09:07.901 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2022-12-03T14:09:07.914 INFO:teuthology.task.internal:Transferring archived files... , 2022-12-03T14:09:07.917 DEBUG:teuthology.misc:Transferring archived files from smithi033:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sjust-2022-12-01_22:28:44-crimson-rados-wip-sjust-testing-2022-11-30-141821-distro-default-smithi/7100485/remote/smithi033 2022-12-03T16:34:12.312 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-1099-g4a79121c (4a79121cfd1ee1750ffed71ede253747b30e6436) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f62aaca1c0e] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5581b57dd0ca] 2022-12-03T16:34:12.314 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:07:54.288 INFO:teuthology.orchestra.run.smithi033.stdout: , 2022-12-03T14:07:54.289 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":222,"fsid":"c5b2e7cc-1e84-45cd-990c-39e1017fbde1","created":"2022-12-03T13:37:20.046999+0000","modified":"2022-12-03T14:07:53.579217+0000","last_up_change":"2022-12-03T13:37:28.012321+0000","last_in_change":"2022-12-03T13:37:22.349737+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-12-03T13:37:29.765153+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-12-03T13:37:30.676877+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2022-12-03T13:37:37.583768+0000","flags":262305,"flags_names":"hashpspool,nopgchange,write_fadvise_dontneed,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"40ba3a2d-f0c0-44fb-86e7-067e28dae341","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6801","nonce":103950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":103950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":103950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6805","nonce":103950}]},"public_addr":"172.21.15.33:6801/103950","cluster_addr":"172.21.15.33:6800/103950","heartbeat_back_addr":"172.21.15.33:6804/103950","heartbeat_front_addr":"172.21.15.33:6805/103950","state":["exists","up"]},{"osd":1,"uuid":"4159bd15-e111-4941-8aaf-10f5848b26bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":103977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6803","nonce":103977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6807","nonce":103977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":103977}]},"public_addr":"172.21.15.33:6802/103977","cluster_addr":"172.21.15.33:6803/103977","heartbeat_back_addr":"172.21.15.33:6807/103977","heartbeat_front_addr":"172.21.15.33:6806/103977","state":["exists","up"]},{"osd":2,"uuid":"3acfb27e-258d-432f-a122-4bea5518efdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6803","nonce":103548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":103548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6807","nonce":103548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":103548}]},"public_addr":"172.21.15.133:6803/103548","cluster_addr":"172.21.15.133:6802/103548","heartbeat_back_addr":"172.21.15.133:6807/103548","heartbeat_front_addr":"172.21.15.133:6806/103548","state":["exists","up"]},{"osd":3,"uuid":"d14bac50-f443-464f-82a7-b5d0406dcf89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":103573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6805","nonce":103573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6809","nonce":103573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":103573}]},"public_addr":"172.21.15.133:6804/103573","cluster_addr":"172.21.15.133:6805/103573","heartbeat_back_addr":"172.21.15.133:6809/103573","heartbeat_front_addr":"172.21.15.133:6808/103573","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2022-12-03T14:07:54.304 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2022-12-03T16:34:12.315 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:07:58.852 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2022-12-03T14:07:58.979+0000 7fa0b835b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2022-12-03T14:07:58.985+0000 7fa0b835b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2022-12-03T16:34:12.316 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:08:04.003 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2022-12-03T13:37:38.178755+0000 time.struct_time(tm_year=2022, tm_mon=12, tm_mday=3, tm_hour=13, tm_min=37, tm_sec=38, tm_wday=5, tm_yday=337, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=12, tm_mday=3, tm_hour=14, tm_min=8, tm_sec=0, tm_wday=5, tm_yday=337, tm_isdst=0) , 2022-12-03T14:08:04.003 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2022-12-03T13:37:31.022058+0000 time.struct_time(tm_year=2022, tm_mon=12, tm_mday=3, tm_hour=13, tm_min=37, tm_sec=31, tm_wday=5, tm_yday=337, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=12, tm_mday=3, tm_hour=14, tm_min=8, tm_sec=0, tm_wday=5, tm_yday=337, tm_isdst=0) , 2022-12-03T14:08:04.004 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2022-12-03T13:37:31.022058+0000 time.struct_time(tm_year=2022, tm_mon=12, tm_mday=3, tm_hour=13, tm_min=37, tm_sec=31, tm_wday=5, tm_yday=337, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=12, tm_mday=3, tm_hour=14, tm_min=8, tm_sec=0, tm_wday=5, tm_yday=337, tm_isdst=0) 2022-12-03T16:34:12.317 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:08:50.434 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar cz -f - -C /var/lib/ceph/crash -- . , 2022-12-03T14:08:50.473 INFO:tasks.ceph:Compressing logs... , 2022-12-03T14:08:50.474 DEBUG:teuthology.orchestra.run.smithi033:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-12-03T16:34:12.318 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-03T14:09:07.901 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2022-12-03T14:09:07.914 INFO:teuthology.task.internal:Transferring archived files... , 2022-12-03T14:09:07.917 DEBUG:teuthology.misc:Transferring archived files from smithi033:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sjust-2022-12-01_22:28:44-crimson-rados-wip-sjust-testing-2022-11-30-141821-distro-default-smithi/7100485/remote/smithi033 2022-12-03T16:34:13.650 INFO:scrape:Found 6 distinct failure reasons 2022-12-03T16:34:13.650 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi159 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4a79121cfd1ee1750ffed71ede253747b30e6436 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2022-12-03T16:34:13.651 INFO:scrape:1 jobs: ['7100474'] 2022-12-03T16:34:13.652 INFO:scrape:suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}'] 2022-12-03T16:34:13.652 INFO:scrape: 2022-12-03T16:34:13.653 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 0 --op snap_remove 0 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2022-12-03T16:34:13.653 INFO:scrape:2 jobs: ['7100482', '7100485'] 2022-12-03T16:34:13.654 INFO:scrape:suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] 2022-12-03T16:34:13.654 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}', 'workloads/write_fadvise_dontneed}'] 2022-12-03T16:34:13.655 INFO:scrape: 2022-12-03T16:34:13.655 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd.sh 2022-12-03T16:34:13.656 INFO:scrape:1 jobs: ['7100475'] 2022-12-03T16:34:13.656 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}'] 2022-12-03T16:34:13.657 INFO:scrape: 2022-12-03T16:34:13.657 INFO:scrape:Failure: Command failed (workunit test rbd/test_librbd_python.sh) on smithi138 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4a79121cfd1ee1750ffed71ede253747b30e6436 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' 2022-12-03T16:34:13.657 INFO:scrape:1 jobs: ['7100484'] 2022-12-03T16:34:13.658 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}'] 2022-12-03T16:34:13.658 INFO:scrape: 2022-12-03T16:34:13.659 INFO:scrape:Failure: reached maximum tries (500) after waiting for 3000 seconds 2022-12-03T16:34:13.659 INFO:scrape:1 jobs: ['7100480'] 2022-12-03T16:34:13.660 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/radosbench}'] 2022-12-03T16:34:13.660 INFO:scrape: 2022-12-03T16:34:13.660 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2022-12-03T16:34:13.661 INFO:scrape:1 jobs: ['7100481'] 2022-12-03T16:34:13.661 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_lock_and_fence}'] 2022-12-03T16:34:13.662 INFO:scrape: