2024-04-01T13:18:31.834 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-01T13:18:31.835 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_matan@teuthology --description crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects} --name matan-2024-04-01_13:06:33-crimson-rados-wip-matanb-crimson-testing-thrash-options-distro-crimson-smithi --verbose --archive /home/teuthworker/archive/matan-2024-04-01_13:06:33-crimson-rados-wip-matanb-crimson-testing-thrash-options-distro-crimson-smithi/7633867 -- /home/teuthworker/archive/matan-2024-04-01_13:06:33-crimson-rados-wip-matanb-crimson-testing-thrash-options-distro-crimson-smithi/7633867/orig.config.yaml 2024-04-01T13:18:31.876 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T13:18:32.003 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-04-01_13:06:33-crimson-rados-wip-matanb-crimson-testing-thrash-options-distro-crimson-smithi/7633867 branch: wip-matanb-crimson-testing-thrash-options description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects} email: null first_in_suite: false job_id: '7633867' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-04-01_13:06:33-crimson-rados-wip-matanb-crimson-testing-thrash-options-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-testing-thrash-options ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED sha1: afa73a9e21b8fb0610f8af7266f1975be37c0a69 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} conf: osd: osd deep scrub update digest min age: 0 install: ceph: flavor: crimson sha1: afa73a9e21b8fb0610f8af7266f1975be37c0a69 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: min_in: 4 workunit: branch: wip-matanb-crimson-testing-thrash-options env: CRIMSON_COMPAT: '1' sha1: afa73a9e21b8fb0610f8af7266f1975be37c0a69 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 5328 sha1: afa73a9e21b8fb0610f8af7266f1975be37c0a69 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-matanb-crimson-testing-thrash-options suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_afa73a9e21b8fb0610f8af7266f1975be37c0a69/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: afa73a9e21b8fb0610f8af7266f1975be37c0a69 targets: smithi059.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC2n7JhWy+x4pN0XflEn9IkPV0yMIou3ZCH758Jn0jgxXkH2bkeblRPAyOihpX2NzK8flS9Dj8L8m9L63z+f1n8= smithi087.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMpS3tU09fadH186AmEtnwjQ3T8VRB9xlaUHIPkeQ8+YtDBUPptD4948Pb6nLVhx5gDd+ofjg3xzvhYmD5OK6vA= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson - thrashosds: ceph_objectstore_tool: false chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - rados: clients: - client.0 objects: 50 op_weights: copy_from: 0 delete: 50 read: 100 rollback: 0 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 pool_snaps: true teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-04-01_13:06:33 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-01T13:18:32.003 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_afa73a9e21b8fb0610f8af7266f1975be37c0a69/qa; will attempt to use it 2024-04-01T13:18:32.004 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_afa73a9e21b8fb0610f8af7266f1975be37c0a69/qa/tasks 2024-04-01T13:18:32.004 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-01T13:18:32.006 INFO:teuthology.task.internal:Checking packages... 2024-04-01T13:18:32.030 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash 'afa73a9e21b8fb0610f8af7266f1975be37c0a69' 2024-04-01T13:18:32.030 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-01T13:18:32.030 INFO:teuthology.packaging:ref: None 2024-04-01T13:18:32.030 INFO:teuthology.packaging:tag: None 2024-04-01T13:18:32.030 INFO:teuthology.packaging:branch: wip-matanb-crimson-testing-thrash-options 2024-04-01T13:18:32.030 INFO:teuthology.packaging:sha1: afa73a9e21b8fb0610f8af7266f1975be37c0a69 2024-04-01T13:18:32.030 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-testing-thrash-options 2024-04-01T13:18:32.266 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2568.gafa73a9e 2024-04-01T13:18:32.268 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-01T13:18:32.275 INFO:teuthology.task.internal:no buildpackages task found 2024-04-01T13:18:32.275 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-01T13:18:32.367 INFO:teuthology.task.internal:Saving configuration 2024-04-01T13:18:32.381 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-01T13:18:32.418 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-01T13:18:32.444 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-01_13:06:33-crimson-rados-wip-matanb-crimson-testing-thrash-options-distro-crimson-smithi/7633867', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-01 13:13:42.559373', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC2n7JhWy+x4pN0XflEn9IkPV0yMIou3ZCH758Jn0jgxXkH2bkeblRPAyOihpX2NzK8flS9Dj8L8m9L63z+f1n8='} 2024-04-01T13:18:32.468 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-01_13:06:33-crimson-rados-wip-matanb-crimson-testing-thrash-options-distro-crimson-smithi/7633867', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-01 13:13:42.561035', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMpS3tU09fadH186AmEtnwjQ3T8VRB9xlaUHIPkeQ8+YtDBUPptD4948Pb6nLVhx5gDd+ofjg3xzvhYmD5OK6vA='} 2024-04-01T13:18:32.468 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-01T13:18:32.474 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-04-01T13:18:32.474 INFO:teuthology.task.internal:roles: ubuntu@smithi087.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-04-01T13:18:32.474 INFO:teuthology.run_tasks:Running task console_log... 2024-04-01T13:18:32.587 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7b7c8f8e50>, signals=[15]) 2024-04-01T13:18:32.588 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-01T13:18:32.593 INFO:teuthology.task.internal:Opening connections... 2024-04-01T13:18:32.594 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2024-04-01T13:18:32.595 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T13:18:32.700 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi087.front.sepia.ceph.com 2024-04-01T13:18:32.701 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T13:18:32.790 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-01T13:18:32.797 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2024-04-01T13:18:32.814 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2024-04-01T13:18:32.814 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2024-04-01T13:18:32.870 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="CentOS Stream" 2024-04-01T13:18:32.871 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="9" 2024-04-01T13:18:32.871 INFO:teuthology.orchestra.run.smithi059.stdout:ID="centos" 2024-04-01T13:18:32.871 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE="rhel fedora" 2024-04-01T13:18:32.871 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="9" 2024-04-01T13:18:32.871 INFO:teuthology.orchestra.run.smithi059.stdout:PLATFORM_ID="platform:el9" 2024-04-01T13:18:32.871 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T13:18:32.871 INFO:teuthology.orchestra.run.smithi059.stdout:ANSI_COLOR="0;31" 2024-04-01T13:18:32.871 INFO:teuthology.orchestra.run.smithi059.stdout:LOGO="fedora-logo-icon" 2024-04-01T13:18:32.871 INFO:teuthology.orchestra.run.smithi059.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T13:18:32.871 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://centos.org/" 2024-04-01T13:18:32.871 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T13:18:32.871 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T13:18:32.871 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T13:18:32.872 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2024-04-01T13:18:32.897 DEBUG:teuthology.orchestra.run.smithi087:> uname -m 2024-04-01T13:18:32.914 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2024-04-01T13:18:32.915 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2024-04-01T13:18:32.969 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="CentOS Stream" 2024-04-01T13:18:32.969 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="9" 2024-04-01T13:18:32.969 INFO:teuthology.orchestra.run.smithi087.stdout:ID="centos" 2024-04-01T13:18:32.969 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE="rhel fedora" 2024-04-01T13:18:32.969 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="9" 2024-04-01T13:18:32.969 INFO:teuthology.orchestra.run.smithi087.stdout:PLATFORM_ID="platform:el9" 2024-04-01T13:18:32.969 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T13:18:32.969 INFO:teuthology.orchestra.run.smithi087.stdout:ANSI_COLOR="0;31" 2024-04-01T13:18:32.969 INFO:teuthology.orchestra.run.smithi087.stdout:LOGO="fedora-logo-icon" 2024-04-01T13:18:32.969 INFO:teuthology.orchestra.run.smithi087.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T13:18:32.969 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://centos.org/" 2024-04-01T13:18:32.970 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T13:18:32.970 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T13:18:32.970 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T13:18:32.970 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2024-04-01T13:18:32.988 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-01T13:18:32.999 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-01T13:18:33.044 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-01T13:18:33.045 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2024-04-01T13:18:33.047 DEBUG:teuthology.orchestra.run.smithi087:> test '!' -e /home/ubuntu/cephtest 2024-04-01T13:18:33.064 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-01T13:18:33.085 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-01T13:18:33.085 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2024-04-01T13:18:33.104 DEBUG:teuthology.orchestra.run.smithi087:> test -z $(ls -A /var/lib/ceph) 2024-04-01T13:18:33.217 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-01T13:18:33.258 INFO:teuthology.run_tasks:Running task kernel... 2024-04-01T13:18:33.288 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-01T13:18:33.288 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-01T13:18:33.288 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-01T13:18:33.289 DEBUG:teuthology.orchestra.run.smithi087:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-01T13:18:33.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T13:18:33.305 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-04-01T13:18:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T13:18:33.307 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-04-01T13:18:33.361 INFO:teuthology.orchestra.run.smithi087.stdout:5.14.0-361.el9.x86_64 2024-04-01T13:18:33.361 INFO:teuthology.task.kernel:Running kernel on smithi087: 5.14.0-361.el9.x86_64 2024-04-01T13:18:33.361 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum install -y kernel 2024-04-01T13:18:33.365 INFO:teuthology.orchestra.run.smithi059.stdout:5.14.0-361.el9.x86_64 2024-04-01T13:18:33.365 INFO:teuthology.task.kernel:Running kernel on smithi059: 5.14.0-361.el9.x86_64 2024-04-01T13:18:33.365 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum install -y kernel 2024-04-01T13:18:35.419 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-04-01T13:18:35.539 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-04-01T13:18:37.111 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-04-01T13:18:37.131 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-04-01T13:18:38.555 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 7.3 MB/s | 8.0 MB 00:01 2024-04-01T13:18:40.210 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.0 MB 00:02 2024-04-01T13:18:40.315 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-04-01T13:18:43.752 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 7.8 MB/s | 19 MB 00:02 2024-04-01T13:18:45.132 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 6.1 MB 00:01 2024-04-01T13:18:46.708 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-01T13:18:48.442 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-01T13:18:53.467 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-01T13:18:55.702 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 754 kB/s | 6.1 MB 00:08 2024-04-01T13:18:55.707 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-01T13:18:55.707 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-01T13:18:55.816 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:18:55.821 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:18:55.821 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-04-01T13:18:55.821 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:18:55.821 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-04-01T13:18:55.821 INFO:teuthology.orchestra.run.smithi087.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-04-01T13:18:55.821 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-04-01T13:18:55.821 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-04-01T13:18:55.821 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-04-01T13:18:55.821 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-04-01T13:18:55.821 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:18:55.821 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-01T13:18:55.821 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:18:55.821 INFO:teuthology.orchestra.run.smithi087.stdout:Install 4 Packages 2024-04-01T13:18:55.822 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:18:55.822 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 76 M 2024-04-01T13:18:55.822 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 123 M 2024-04-01T13:18:55.822 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-01T13:18:55.993 INFO:teuthology.orchestra.run.smithi087.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-04-01T13:18:56.727 INFO:teuthology.orchestra.run.smithi087.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 17 MB/s | 15 MB 00:00 2024-04-01T13:18:57.269 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-04-01T13:18:57.502 INFO:teuthology.orchestra.run.smithi087.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 20 MB/s | 34 MB 00:01 2024-04-01T13:18:59.011 INFO:teuthology.orchestra.run.smithi087.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 9.2 MB/s | 28 MB 00:03 2024-04-01T13:18:59.014 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:18:59.015 INFO:teuthology.orchestra.run.smithi087.stdout:Total 24 MB/s | 76 MB 00:03 2024-04-01T13:18:59.015 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-01T13:18:59.377 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-01T13:18:59.377 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-01T13:19:00.023 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 21 MB 00:02 2024-04-01T13:19:00.210 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-01T13:19:00.211 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-01T13:19:02.525 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-01T13:19:03.116 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-04-01T13:19:03.211 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T13:19:05.162 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-01T13:19:05.943 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T13:19:06.081 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T13:19:07.519 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-01T13:19:07.519 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-01T13:19:07.618 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:19:07.622 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:19:07.622 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-04-01T13:19:07.623 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:19:07.623 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-04-01T13:19:07.623 INFO:teuthology.orchestra.run.smithi059.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-04-01T13:19:07.623 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-04-01T13:19:07.623 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-04-01T13:19:07.623 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-04-01T13:19:07.623 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-04-01T13:19:07.623 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:19:07.623 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-04-01T13:19:07.623 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:19:07.623 INFO:teuthology.orchestra.run.smithi059.stdout:Install 4 Packages 2024-04-01T13:19:07.623 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:19:07.624 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 76 M 2024-04-01T13:19:07.624 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 123 M 2024-04-01T13:19:07.624 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-04-01T13:19:07.781 INFO:teuthology.orchestra.run.smithi059.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-04-01T13:19:08.414 INFO:teuthology.orchestra.run.smithi059.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 19 MB/s | 15 MB 00:00 2024-04-01T13:19:08.989 INFO:teuthology.orchestra.run.smithi059.stdout:(3/4): kernel-modules-core-5.14.0-434.el9.x86_6 23 MB/s | 28 MB 00:01 2024-04-01T13:19:09.439 INFO:teuthology.orchestra.run.smithi059.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 19 MB/s | 34 MB 00:01 2024-04-01T13:19:09.442 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:19:09.443 INFO:teuthology.orchestra.run.smithi059.stdout:Total 42 MB/s | 76 MB 00:01 2024-04-01T13:19:09.443 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-04-01T13:19:09.884 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-04-01T13:19:09.884 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-04-01T13:19:10.770 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-04-01T13:19:10.771 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-04-01T13:19:10.777 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T13:19:10.813 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-04-01T13:19:13.182 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-04-01T13:19:13.819 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-04-01T13:19:13.907 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T13:19:15.513 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T13:19:16.786 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T13:19:16.926 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T13:19:22.001 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T13:19:22.042 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-04-01T13:19:27.143 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T13:19:44.882 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T13:19:44.882 INFO:teuthology.orchestra.run.smithi087.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-01T13:19:44.882 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:19:44.924 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-04-01T13:19:45.284 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-04-01T13:19:45.284 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-04-01T13:19:45.284 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T13:19:45.284 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T13:19:45.683 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T13:19:45.684 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:19:45.684 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-04-01T13:19:45.684 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-5.14.0-434.el9.x86_64 2024-04-01T13:19:45.684 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-04-01T13:19:45.684 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-04-01T13:19:45.684 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-04-01T13:19:45.684 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:19:45.684 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:19:46.001 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-04-01T13:19:46.114 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-01T13:19:46.114 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-01T13:19:46.114 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-04-01T13:19:46.114 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-01T13:19:46.114 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-01T13:19:46.114 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-04-01T13:19:46.115 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi087.front.sepia.ceph.com, path=None, version=distro) 2024-04-01T13:19:46.115 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-04-01T13:19:46.195 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-01T13:19:46.195 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-01T13:19:46.195 DEBUG:teuthology.orchestra.run.smithi087:> sudo rpm -qi grub2-tools 2024-04-01T13:19:46.240 INFO:teuthology.orchestra.run.smithi087.stdout:Name : grub2-tools 2024-04-01T13:19:46.240 INFO:teuthology.orchestra.run.smithi087.stdout:Epoch : 1 2024-04-01T13:19:46.240 INFO:teuthology.orchestra.run.smithi087.stdout:Version : 2.06 2024-04-01T13:19:46.240 INFO:teuthology.orchestra.run.smithi087.stdout:Release : 68.el9 2024-04-01T13:19:46.240 INFO:teuthology.orchestra.run.smithi087.stdout:Architecture: x86_64 2024-04-01T13:19:46.240 INFO:teuthology.orchestra.run.smithi087.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-01T13:19:46.240 INFO:teuthology.orchestra.run.smithi087.stdout:Group : Unspecified 2024-04-01T13:19:46.241 INFO:teuthology.orchestra.run.smithi087.stdout:Size : 8264254 2024-04-01T13:19:46.241 INFO:teuthology.orchestra.run.smithi087.stdout:License : GPLv3+ 2024-04-01T13:19:46.241 INFO:teuthology.orchestra.run.smithi087.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-01T13:19:46.241 INFO:teuthology.orchestra.run.smithi087.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-01T13:19:46.241 INFO:teuthology.orchestra.run.smithi087.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-01T13:19:46.241 INFO:teuthology.orchestra.run.smithi087.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-01T13:19:46.241 INFO:teuthology.orchestra.run.smithi087.stdout:Packager : builder@centos.org 2024-04-01T13:19:46.241 INFO:teuthology.orchestra.run.smithi087.stdout:Vendor : CentOS 2024-04-01T13:19:46.241 INFO:teuthology.orchestra.run.smithi087.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-01T13:19:46.241 INFO:teuthology.orchestra.run.smithi087.stdout:Summary : Support tools for GRUB. 2024-04-01T13:19:46.241 INFO:teuthology.orchestra.run.smithi087.stdout:Description : 2024-04-01T13:19:46.241 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:19:46.241 INFO:teuthology.orchestra.run.smithi087.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-01T13:19:46.241 INFO:teuthology.orchestra.run.smithi087.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-01T13:19:46.241 INFO:teuthology.orchestra.run.smithi087.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-01T13:19:46.242 INFO:teuthology.orchestra.run.smithi087.stdout:hardware devices. 2024-04-01T13:19:46.242 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:19:46.242 INFO:teuthology.orchestra.run.smithi087.stdout:This subpackage provides tools for support of all platforms. 2024-04-01T13:19:46.243 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-01T13:19:46.243 INFO:teuthology.task.kernel:Updating grub on smithi087 to boot 5.14.0-434.el9.x86_64 2024-04-01T13:19:46.243 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-01T13:19:46.855 INFO:teuthology.orchestra.run.smithi087.stderr:Generating grub configuration file ... 2024-04-01T13:19:48.159 INFO:teuthology.orchestra.run.smithi087.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-01T13:19:48.194 INFO:teuthology.orchestra.run.smithi087.stderr:done 2024-04-01T13:19:48.196 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-04-01T13:19:48.211 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.GsSnxJwUwX 2024-04-01T13:19:48.211 DEBUG:teuthology.orchestra.run.smithi087:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.GsSnxJwUwX 2024-04-01T13:19:48.278 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.GsSnxJwUwX 2024-04-01T13:19:48.416 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.GsSnxJwUwX is 6KB 2024-04-01T13:19:48.465 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.GsSnxJwUwX 2024-04-01T13:19:48.479 DEBUG:teuthology.orchestra.run.smithi087:> sudo /bin/ls /boot/loader/entries || true 2024-04-01T13:19:48.544 INFO:teuthology.orchestra.run.smithi087.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-01T13:19:48.545 INFO:teuthology.orchestra.run.smithi087.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-01T13:19:48.545 INFO:teuthology.orchestra.run.smithi087.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-01T13:19:48.545 INFO:teuthology.orchestra.run.smithi087.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-04-01T13:19:48.547 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-04-01T13:19:48.680 DEBUG:teuthology.orchestra.run.smithi087:> sudo shutdown -r now 2024-04-01T13:19:58.224 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T13:19:58.224 INFO:teuthology.orchestra.run.smithi059.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-01T13:19:58.224 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:19:58.268 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-04-01T13:19:58.602 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-04-01T13:19:58.602 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-04-01T13:19:58.602 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T13:19:58.602 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T13:19:59.022 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T13:19:59.023 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:19:59.023 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-04-01T13:19:59.023 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-5.14.0-434.el9.x86_64 2024-04-01T13:19:59.023 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-04-01T13:19:59.023 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-04-01T13:19:59.023 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-04-01T13:19:59.023 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:19:59.023 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:19:59.364 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2024-04-01T13:19:59.464 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-01T13:19:59.464 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-01T13:19:59.464 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-04-01T13:19:59.464 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-01T13:19:59.464 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-01T13:19:59.464 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-01T13:19:59.464 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi059.front.sepia.ceph.com, path=None, version=distro) 2024-04-01T13:19:59.464 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2024-04-01T13:19:59.548 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-01T13:19:59.548 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-01T13:19:59.548 DEBUG:teuthology.orchestra.run.smithi059:> sudo rpm -qi grub2-tools 2024-04-01T13:19:59.589 INFO:teuthology.orchestra.run.smithi059.stdout:Name : grub2-tools 2024-04-01T13:19:59.589 INFO:teuthology.orchestra.run.smithi059.stdout:Epoch : 1 2024-04-01T13:19:59.589 INFO:teuthology.orchestra.run.smithi059.stdout:Version : 2.06 2024-04-01T13:19:59.589 INFO:teuthology.orchestra.run.smithi059.stdout:Release : 68.el9 2024-04-01T13:19:59.589 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: x86_64 2024-04-01T13:19:59.589 INFO:teuthology.orchestra.run.smithi059.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-01T13:19:59.589 INFO:teuthology.orchestra.run.smithi059.stdout:Group : Unspecified 2024-04-01T13:19:59.589 INFO:teuthology.orchestra.run.smithi059.stdout:Size : 8264254 2024-04-01T13:19:59.589 INFO:teuthology.orchestra.run.smithi059.stdout:License : GPLv3+ 2024-04-01T13:19:59.590 INFO:teuthology.orchestra.run.smithi059.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-01T13:19:59.590 INFO:teuthology.orchestra.run.smithi059.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-01T13:19:59.590 INFO:teuthology.orchestra.run.smithi059.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-01T13:19:59.590 INFO:teuthology.orchestra.run.smithi059.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-01T13:19:59.590 INFO:teuthology.orchestra.run.smithi059.stdout:Packager : builder@centos.org 2024-04-01T13:19:59.590 INFO:teuthology.orchestra.run.smithi059.stdout:Vendor : CentOS 2024-04-01T13:19:59.590 INFO:teuthology.orchestra.run.smithi059.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-01T13:19:59.590 INFO:teuthology.orchestra.run.smithi059.stdout:Summary : Support tools for GRUB. 2024-04-01T13:19:59.590 INFO:teuthology.orchestra.run.smithi059.stdout:Description : 2024-04-01T13:19:59.590 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:19:59.590 INFO:teuthology.orchestra.run.smithi059.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-01T13:19:59.590 INFO:teuthology.orchestra.run.smithi059.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-01T13:19:59.590 INFO:teuthology.orchestra.run.smithi059.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-01T13:19:59.590 INFO:teuthology.orchestra.run.smithi059.stdout:hardware devices. 2024-04-01T13:19:59.590 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:19:59.591 INFO:teuthology.orchestra.run.smithi059.stdout:This subpackage provides tools for support of all platforms. 2024-04-01T13:19:59.591 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-01T13:19:59.592 INFO:teuthology.task.kernel:Updating grub on smithi059 to boot 5.14.0-434.el9.x86_64 2024-04-01T13:19:59.592 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-01T13:20:00.187 INFO:teuthology.orchestra.run.smithi059.stderr:Generating grub configuration file ... 2024-04-01T13:20:01.555 INFO:teuthology.orchestra.run.smithi059.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-01T13:20:01.592 INFO:teuthology.orchestra.run.smithi059.stderr:done 2024-04-01T13:20:01.594 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-04-01T13:20:01.609 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.dqHZtyXm94 2024-04-01T13:20:01.610 DEBUG:teuthology.orchestra.run.smithi059:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dqHZtyXm94 2024-04-01T13:20:01.683 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.dqHZtyXm94 2024-04-01T13:20:01.825 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.dqHZtyXm94 is 6KB 2024-04-01T13:20:01.874 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.dqHZtyXm94 2024-04-01T13:20:01.888 DEBUG:teuthology.orchestra.run.smithi059:> sudo /bin/ls /boot/loader/entries || true 2024-04-01T13:20:01.957 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-01T13:20:01.957 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-01T13:20:01.957 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-01T13:20:01.957 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-04-01T13:20:01.959 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-04-01T13:20:02.117 DEBUG:teuthology.orchestra.run.smithi059:> sudo shutdown -r now 2024-04-01T13:20:18.698 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-04-01T13:20:18.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-04-01T13:20:18.700 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T13:20:32.133 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-01T13:20:32.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-04-01T13:20:32.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T13:20:50.463 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.59 2024-04-01T13:20:59.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-04-01T13:20:59.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T13:21:18.713 DEBUG:teuthology.orchestra.remote:timed out 2024-04-01T13:21:27.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-04-01T13:21:27.723 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T13:21:31.122 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-04-01T13:21:31.591 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-04-01T13:21:31.591 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-434.el9.x86_64"... 2024-04-01T13:21:31.591 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-04-01T13:21:31.606 INFO:teuthology.orchestra.run.smithi087.stdout:5.14.0-434.el9.x86_64 2024-04-01T13:21:31.606 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-04-01T13:21:31.606 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-01T13:21:31.606 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-01T13:21:32.607 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-04-01T13:21:32.608 DEBUG:teuthology.orchestra.run.smithi087:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-01T13:21:32.679 INFO:teuthology.orchestra.run.smithi087.stdout:ttyS1 2024-04-01T13:21:32.712 DEBUG:teuthology.parallel:result is None 2024-04-01T13:21:59.474 DEBUG:teuthology.orchestra.remote:timed out 2024-04-01T13:22:11.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-04-01T13:22:11.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T13:22:11.795 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-04-01T13:22:12.169 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-04-01T13:22:12.170 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-04-01T13:22:12.170 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-04-01T13:22:12.184 INFO:teuthology.orchestra.run.smithi059.stdout:5.14.0-434.el9.x86_64 2024-04-01T13:22:12.184 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-04-01T13:22:12.184 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-01T13:22:12.184 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-01T13:22:13.185 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-01T13:22:13.185 DEBUG:teuthology.orchestra.run.smithi059:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-01T13:22:13.240 INFO:teuthology.orchestra.run.smithi059.stdout:ttyS1 2024-04-01T13:22:13.283 DEBUG:teuthology.parallel:result is None 2024-04-01T13:22:13.283 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-01T13:22:13.291 INFO:teuthology.task.internal:Creating test directory... 2024-04-01T13:22:13.291 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-01T13:22:13.293 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-01T13:22:13.313 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-01T13:22:13.339 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-01T13:22:13.414 INFO:teuthology.task.internal:Creating archive directory... 2024-04-01T13:22:13.415 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-01T13:22:13.417 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-01T13:22:13.451 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-01T13:22:13.496 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-01T13:22:13.496 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-01T13:22:13.499 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-01T13:22:13.535 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T13:22:13.538 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T13:22:13.549 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T13:22:13.552 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T13:22:13.554 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-01T13:22:13.563 INFO:teuthology.task.internal:Configuring sudo... 2024-04-01T13:22:13.563 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-01T13:22:13.594 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-01T13:22:13.625 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-01T13:22:13.635 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-01T13:22:13.635 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-01T13:22:13.664 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-01T13:22:13.681 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-01T13:22:13.763 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-01T13:22:13.814 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-01T13:22:13.896 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-01T13:22:13.942 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-01T13:22:13.942 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-01T13:22:14.010 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-01T13:22:14.059 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-01T13:22:14.110 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-01T13:22:14.190 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-01T13:22:14.238 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-01T13:22:14.238 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-01T13:22:14.307 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2024-04-01T13:22:14.309 DEBUG:teuthology.orchestra.run.smithi087:> sudo service rsyslog restart 2024-04-01T13:22:14.353 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-01T13:22:14.403 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-01T13:22:14.738 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-01T13:22:14.746 INFO:teuthology.task.internal:Starting timer... 2024-04-01T13:22:14.746 INFO:teuthology.run_tasks:Running task pcp... 2024-04-01T13:22:14.808 INFO:teuthology.run_tasks:Running task selinux... 2024-04-01T13:22:14.885 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-01T13:22:14.885 DEBUG:teuthology.orchestra.run.smithi059:> sudo service auditd rotate 2024-04-01T13:22:14.952 INFO:teuthology.orchestra.run.smithi059.stdout:Rotating logs: 2024-04-01T13:22:14.954 DEBUG:teuthology.orchestra.run.smithi087:> sudo service auditd rotate 2024-04-01T13:22:15.018 INFO:teuthology.orchestra.run.smithi087.stdout:Rotating logs: 2024-04-01T13:22:15.020 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-01T13:22:15.021 DEBUG:teuthology.orchestra.run.smithi059:> /usr/sbin/getenforce 2024-04-01T13:22:15.041 INFO:teuthology.orchestra.run.smithi059.stdout:Permissive 2024-04-01T13:22:15.041 DEBUG:teuthology.orchestra.run.smithi087:> /usr/sbin/getenforce 2024-04-01T13:22:15.063 INFO:teuthology.orchestra.run.smithi087.stdout:Permissive 2024-04-01T13:22:15.063 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi059.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi087.front.sepia.ceph.com': 'permissive'} 2024-04-01T13:22:15.063 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-01T13:22:15.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T13:22:15.108 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-01T13:22:15.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T13:22:15.136 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-01T13:22:15.137 DEBUG:teuthology.orchestra.run.smithi059:> sudo /usr/sbin/setenforce permissive 2024-04-01T13:22:15.184 DEBUG:teuthology.orchestra.run.smithi087:> sudo /usr/sbin/setenforce permissive 2024-04-01T13:22:15.215 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-01T13:22:15.224 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-01T13:22:15.230 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-01T13:22:15.372 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-01T13:22:15.388 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-01T13:22:15.389 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi059.front.sepia.ceph.com,smithi087.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-01T13:26:36.003 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi059.front.sepia.ceph.com'), Remote(name='ubuntu@smithi087.front.sepia.ceph.com')] 2024-04-01T13:26:36.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-04-01T13:26:36.006 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T13:26:36.094 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-04-01T13:26:36.177 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-04-01T13:26:36.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-04-01T13:26:36.178 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T13:26:36.267 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-04-01T13:26:36.344 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-04-01T13:26:36.345 INFO:teuthology.run_tasks:Running task clock... 2024-04-01T13:26:36.359 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-01T13:26:36.359 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-01T13:26:36.360 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-01T13:26:36.362 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-01T13:26:36.362 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-01T13:26:36.393 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-01T13:26:36.410 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-01T13:26:36.415 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-01T13:26:36.432 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-01T13:26:36.441 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: ntpd: command not found 2024-04-01T13:26:36.455 INFO:teuthology.orchestra.run.smithi059.stdout:506 Cannot talk to daemon 2024-04-01T13:26:36.461 INFO:teuthology.orchestra.run.smithi087.stderr:sudo: ntpd: command not found 2024-04-01T13:26:36.472 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-01T13:26:36.474 INFO:teuthology.orchestra.run.smithi087.stdout:506 Cannot talk to daemon 2024-04-01T13:26:36.488 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-01T13:26:36.491 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-01T13:26:36.507 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-01T13:26:36.541 INFO:teuthology.orchestra.run.smithi059.stderr:bash: line 1: ntpq: command not found 2024-04-01T13:26:36.546 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-01T13:26:36.546 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2024-04-01T13:26:36.546 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T13:26:36.546 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T13:26:36.546 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T13:26:36.546 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T13:26:36.560 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2024-04-01T13:26:36.563 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-01T13:26:36.563 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-04-01T13:26:36.563 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T13:26:36.563 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T13:26:36.564 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T13:26:36.564 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T13:26:36.564 INFO:teuthology.run_tasks:Running task install... 2024-04-01T13:26:36.572 DEBUG:teuthology.task.install:project ceph 2024-04-01T13:26:36.572 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': 'afa73a9e21b8fb0610f8af7266f1975be37c0a69'}} 2024-04-01T13:26:36.572 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': 'afa73a9e21b8fb0610f8af7266f1975be37c0a69'} 2024-04-01T13:26:36.572 INFO:teuthology.task.install:Using flavor: crimson 2024-04-01T13:26:36.577 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-01T13:26:36.577 INFO:teuthology.task.install:extra packages: [] 2024-04-01T13:26:36.578 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'afa73a9e21b8fb0610f8af7266f1975be37c0a69', 'tag': None, 'wait_for_package': False} 2024-04-01T13:26:36.578 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=afa73a9e21b8fb0610f8af7266f1975be37c0a69 2024-04-01T13:26:36.580 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'afa73a9e21b8fb0610f8af7266f1975be37c0a69', 'tag': None, 'wait_for_package': False} 2024-04-01T13:26:36.580 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=afa73a9e21b8fb0610f8af7266f1975be37c0a69 2024-04-01T13:26:36.827 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-thrash-options/afa73a9e21b8fb0610f8af7266f1975be37c0a69/centos/9/flavors/crimson/ 2024-04-01T13:26:36.827 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2568.gafa73a9e 2024-04-01T13:26:36.835 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-thrash-options/afa73a9e21b8fb0610f8af7266f1975be37c0a69/centos/9/flavors/crimson/ 2024-04-01T13:26:36.835 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2568.gafa73a9e 2024-04-01T13:26:37.058 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-thrash-options/afa73a9e21b8fb0610f8af7266f1975be37c0a69/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-thrash-options/afa73a9e21b8fb0610f8af7266f1975be37c0a69/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-thrash-options/afa73a9e21b8fb0610f8af7266f1975be37c0a69/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-01T13:26:37.058 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-01T13:26:37.058 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-01T13:26:37.072 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-thrash-options/afa73a9e21b8fb0610f8af7266f1975be37c0a69/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-thrash-options/afa73a9e21b8fb0610f8af7266f1975be37c0a69/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-thrash-options/afa73a9e21b8fb0610f8af7266f1975be37c0a69/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-01T13:26:37.072 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-01T13:26:37.072 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-01T13:26:37.087 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-01T13:26:37.087 DEBUG:teuthology.orchestra.run.smithi059:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/afa73a9e21b8fb0610f8af7266f1975be37c0a69/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-01T13:26:37.100 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-01T13:26:37.101 DEBUG:teuthology.orchestra.run.smithi087:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/afa73a9e21b8fb0610f8af7266f1975be37c0a69/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-01T13:26:37.156 DEBUG:teuthology.orchestra.run.smithi059:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-01T13:26:37.169 DEBUG:teuthology.orchestra.run.smithi087:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-01T13:26:37.238 DEBUG:teuthology.orchestra.run.smithi059:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-01T13:26:37.249 DEBUG:teuthology.orchestra.run.smithi087:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-01T13:26:37.304 INFO:teuthology.orchestra.run.smithi059.stdout:check_obsoletes = 1 2024-04-01T13:26:37.305 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum clean all 2024-04-01T13:26:37.316 INFO:teuthology.orchestra.run.smithi087.stdout:check_obsoletes = 1 2024-04-01T13:26:37.317 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum clean all 2024-04-01T13:26:37.546 INFO:teuthology.orchestra.run.smithi059.stdout:74 files removed 2024-04-01T13:26:37.569 INFO:teuthology.orchestra.run.smithi087.stdout:74 files removed 2024-04-01T13:26:37.573 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-radosgw 2024-04-01T13:26:37.595 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-radosgw 2024-04-01T13:26:38.344 INFO:teuthology.orchestra.run.smithi059.stdout:ceph packages for x86_64 187 kB/s | 87 kB 00:00 2024-04-01T13:26:38.348 INFO:teuthology.orchestra.run.smithi087.stdout:ceph packages for x86_64 191 kB/s | 87 kB 00:00 2024-04-01T13:26:38.670 INFO:teuthology.orchestra.run.smithi059.stdout:ceph noarch packages 42 kB/s | 12 kB 00:00 2024-04-01T13:26:38.782 INFO:teuthology.orchestra.run.smithi087.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-04-01T13:26:39.063 INFO:teuthology.orchestra.run.smithi059.stdout:ceph source packages 5.8 kB/s | 2.0 kB 00:00 2024-04-01T13:26:39.099 INFO:teuthology.orchestra.run.smithi087.stdout:ceph source packages 7.3 kB/s | 2.0 kB 00:00 2024-04-01T13:26:39.548 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-04-01T13:26:39.827 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS-9 - AppStream 9.7 MB/s | 6.7 MB 00:00 2024-04-01T13:26:40.998 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-04-01T13:26:41.806 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-04-01T13:26:42.253 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS-9 - BaseOS 1.6 MB/s | 2.0 MB 00:01 2024-04-01T13:26:43.401 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 9.7 MB/s | 8.0 MB 00:00 2024-04-01T13:26:43.815 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-01T13:26:45.219 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-01T13:26:47.816 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-04-01T13:26:49.275 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 16 kB 00:00 2024-04-01T13:26:49.295 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-04-01T13:26:50.427 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-04-01T13:26:50.930 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-01T13:26:55.552 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-01T13:26:57.030 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 3.5 MB/s | 21 MB 00:05 2024-04-01T13:26:58.207 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:26:58.210 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:26:58.210 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-04-01T13:26:58.210 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:26:58.210 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-04-01T13:26:58.210 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 9.6 M 2024-04-01T13:26:58.210 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrading: 2024-04-01T13:26:58.210 INFO:teuthology.orchestra.run.smithi059.stdout: librados2 x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 3.4 M 2024-04-01T13:26:58.210 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1 x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 3.7 M 2024-04-01T13:26:58.211 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-04-01T13:26:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: boost-program-options 2024-04-01T13:26:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-01T13:26:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 5.9 M 2024-04-01T13:26:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 19 M 2024-04-01T13:26:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 25 k 2024-04-01T13:26:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-01T13:26:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-01T13:26:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-01T13:26:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2 x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 797 k 2024-04-01T13:26:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-01T13:26:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-01T13:26:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-01T13:26:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1 x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 548 k 2024-04-01T13:26:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-01T13:26:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2 x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 5.0 M 2024-04-01T13:26:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-01T13:26:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-01T13:26:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-01T13:26:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse 2024-04-01T13:26:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 45 k 2024-04-01T13:26:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 134 k 2024-04-01T13:26:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 159 k 2024-04-01T13:26:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-01T13:26:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 315 k 2024-04-01T13:26:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 293 k 2024-04-01T13:26:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 99 k 2024-04-01T13:26:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-01T13:26:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-01T13:26:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-01T13:26:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-01T13:26:58.213 INFO:teuthology.orchestra.run.smithi059.stdout:Installing weak dependencies: 2024-04-01T13:26:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-01T13:26:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:26:58.213 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-04-01T13:26:58.213 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:26:58.213 INFO:teuthology.orchestra.run.smithi059.stdout:Install 30 Packages 2024-04-01T13:26:58.213 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrade 2 Packages 2024-04-01T13:26:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:26:58.214 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 58 M 2024-04-01T13:26:58.214 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-04-01T13:26:59.403 INFO:teuthology.orchestra.run.smithi059.stdout:(1/32): ceph-radosgw-19.0.0-2568.gafa73a9e.el9. 9.3 MB/s | 9.6 MB 00:01 2024-04-01T13:26:59.462 INFO:teuthology.orchestra.run.smithi059.stdout:(2/32): ceph-selinux-19.0.0-2568.gafa73a9e.el9. 431 kB/s | 25 kB 00:00 2024-04-01T13:26:59.562 INFO:teuthology.orchestra.run.smithi059.stdout:(3/32): libcephfs2-19.0.0-2568.gafa73a9e.el9.x8 7.8 MB/s | 797 kB 00:00 2024-04-01T13:26:59.654 INFO:teuthology.orchestra.run.smithi059.stdout:(4/32): libradosstriper1-19.0.0-2568.gafa73a9e. 5.9 MB/s | 548 kB 00:00 2024-04-01T13:26:59.954 INFO:teuthology.orchestra.run.smithi059.stdout:(5/32): ceph-common-19.0.0-2568.gafa73a9e.el9.x 12 MB/s | 19 MB 00:01 2024-04-01T13:27:00.013 INFO:teuthology.orchestra.run.smithi059.stdout:(6/32): python3-ceph-argparse-19.0.0-2568.gafa7 777 kB/s | 45 kB 00:00 2024-04-01T13:27:00.088 INFO:teuthology.orchestra.run.smithi059.stdout:(7/32): librgw2-19.0.0-2568.gafa73a9e.el9.x86_6 12 MB/s | 5.0 MB 00:00 2024-04-01T13:27:00.105 INFO:teuthology.orchestra.run.smithi059.stdout:(8/32): python3-ceph-common-19.0.0-2568.gafa73a 1.4 MB/s | 134 kB 00:00 2024-04-01T13:27:00.130 INFO:teuthology.orchestra.run.smithi059.stdout:(9/32): python3-cephfs-19.0.0-2568.gafa73a9e.el 3.7 MB/s | 159 kB 00:00 2024-04-01T13:27:00.164 INFO:teuthology.orchestra.run.smithi059.stdout:(10/32): python3-rados-19.0.0-2568.gafa73a9e.el 5.3 MB/s | 315 kB 00:00 2024-04-01T13:27:00.189 INFO:teuthology.orchestra.run.smithi059.stdout:(11/32): python3-rbd-19.0.0-2568.gafa73a9e.el9. 4.9 MB/s | 293 kB 00:00 2024-04-01T13:27:00.214 INFO:teuthology.orchestra.run.smithi059.stdout:(12/32): python3-rgw-19.0.0-2568.gafa73a9e.el9. 1.9 MB/s | 99 kB 00:00 2024-04-01T13:27:00.406 INFO:teuthology.orchestra.run.smithi059.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 488 kB/s | 106 kB 00:00 2024-04-01T13:27:00.457 INFO:teuthology.orchestra.run.smithi059.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 741 kB/s | 159 kB 00:00 2024-04-01T13:27:00.473 INFO:teuthology.orchestra.run.smithi059.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 661 kB/s | 44 kB 00:00 2024-04-01T13:27:00.557 INFO:teuthology.orchestra.run.smithi059.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 661 kB 00:00 2024-04-01T13:27:00.582 INFO:teuthology.orchestra.run.smithi059.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.5 MB/s | 282 kB 00:00 2024-04-01T13:27:00.608 INFO:teuthology.orchestra.run.smithi059.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 830 kB/s | 41 kB 00:00 2024-04-01T13:27:00.641 INFO:teuthology.orchestra.run.smithi059.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 220 kB 00:00 2024-04-01T13:27:00.667 INFO:teuthology.orchestra.run.smithi059.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 596 kB/s | 35 kB 00:00 2024-04-01T13:27:00.692 INFO:teuthology.orchestra.run.smithi059.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 923 kB/s | 46 kB 00:00 2024-04-01T13:27:00.793 INFO:teuthology.orchestra.run.smithi059.stdout:(22/32): ceph-base-19.0.0-2568.gafa73a9e.el9.x8 2.4 MB/s | 5.9 MB 00:02 2024-04-01T13:27:01.218 INFO:teuthology.orchestra.run.smithi059.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 70 kB/s | 26 kB 00:00 2024-04-01T13:27:01.626 INFO:teuthology.orchestra.run.smithi059.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 322 kB/s | 309 kB 00:00 2024-04-01T13:27:02.005 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-01T13:27:02.185 INFO:teuthology.orchestra.run.smithi059.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 121 kB/s | 67 kB 00:00 2024-04-01T13:27:02.252 INFO:teuthology.orchestra.run.smithi059.stdout:(26/32): liboath-2.6.7-2.el9.x86_64.rpm 47 kB/s | 49 kB 00:01 2024-04-01T13:27:02.444 INFO:teuthology.orchestra.run.smithi059.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2024-04-01T13:27:03.810 INFO:teuthology.orchestra.run.smithi059.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 516 kB/s | 838 kB 00:01 2024-04-01T13:27:03.877 INFO:teuthology.orchestra.run.smithi059.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 1.1 MB/s | 1.6 MB 00:01 2024-04-01T13:27:03.961 INFO:teuthology.orchestra.run.smithi059.stdout:(30/32): libarrow-9.0.0-10.el9.x86_64.rpm 1.4 MB/s | 4.4 MB 00:03 2024-04-01T13:27:04.219 INFO:teuthology.orchestra.run.smithi059.stdout:(31/32): librbd1-19.0.0-2568.gafa73a9e.el9.x86_ 11 MB/s | 3.7 MB 00:00 2024-04-01T13:27:04.486 INFO:teuthology.orchestra.run.smithi059.stdout:(32/32): librados2-19.0.0-2568.gafa73a9e.el9.x8 5.0 MB/s | 3.4 MB 00:00 2024-04-01T13:27:04.494 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:27:04.494 INFO:teuthology.orchestra.run.smithi059.stdout:Total 9.2 MB/s | 58 MB 00:06 2024-04-01T13:27:04.494 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-04-01T13:27:04.537 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-04-01T13:27:04.538 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-04-01T13:27:04.588 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:27:04.604 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:27:04.604 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-04-01T13:27:04.604 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:27:04.604 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-04-01T13:27:04.604 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 9.6 M 2024-04-01T13:27:04.604 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrading: 2024-04-01T13:27:04.604 INFO:teuthology.orchestra.run.smithi087.stdout: librados2 x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 3.4 M 2024-04-01T13:27:04.605 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1 x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 3.7 M 2024-04-01T13:27:04.605 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-04-01T13:27:04.605 INFO:teuthology.orchestra.run.smithi087.stdout: boost-program-options 2024-04-01T13:27:04.605 INFO:teuthology.orchestra.run.smithi087.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-01T13:27:04.605 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 5.9 M 2024-04-01T13:27:04.605 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 19 M 2024-04-01T13:27:04.605 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 25 k 2024-04-01T13:27:04.605 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-01T13:27:04.605 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-01T13:27:04.605 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-01T13:27:04.605 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2 x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 797 k 2024-04-01T13:27:04.605 INFO:teuthology.orchestra.run.smithi087.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-01T13:27:04.605 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-01T13:27:04.606 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-01T13:27:04.606 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1 x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 548 k 2024-04-01T13:27:04.606 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-01T13:27:04.606 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2 x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 5.0 M 2024-04-01T13:27:04.606 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-01T13:27:04.606 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-01T13:27:04.606 INFO:teuthology.orchestra.run.smithi087.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-01T13:27:04.606 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse 2024-04-01T13:27:04.606 INFO:teuthology.orchestra.run.smithi087.stdout: x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 45 k 2024-04-01T13:27:04.606 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 134 k 2024-04-01T13:27:04.606 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 159 k 2024-04-01T13:27:04.606 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-01T13:27:04.606 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 315 k 2024-04-01T13:27:04.606 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 293 k 2024-04-01T13:27:04.607 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 99 k 2024-04-01T13:27:04.607 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-01T13:27:04.607 INFO:teuthology.orchestra.run.smithi087.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-01T13:27:04.607 INFO:teuthology.orchestra.run.smithi087.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-01T13:27:04.607 INFO:teuthology.orchestra.run.smithi087.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-01T13:27:04.607 INFO:teuthology.orchestra.run.smithi087.stdout:Installing weak dependencies: 2024-04-01T13:27:04.607 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-01T13:27:04.607 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:27:04.607 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-01T13:27:04.607 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:27:04.607 INFO:teuthology.orchestra.run.smithi087.stdout:Install 30 Packages 2024-04-01T13:27:04.607 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrade 2 Packages 2024-04-01T13:27:04.607 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:27:04.608 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 58 M 2024-04-01T13:27:04.608 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-01T13:27:05.023 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-04-01T13:27:05.023 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-04-01T13:27:06.089 INFO:teuthology.orchestra.run.smithi087.stdout:(1/32): ceph-radosgw-19.0.0-2568.gafa73a9e.el9. 8.4 MB/s | 9.6 MB 00:01 2024-04-01T13:27:06.131 INFO:teuthology.orchestra.run.smithi087.stdout:(2/32): ceph-selinux-19.0.0-2568.gafa73a9e.el9. 604 kB/s | 25 kB 00:00 2024-04-01T13:27:06.273 INFO:teuthology.orchestra.run.smithi087.stdout:(3/32): libcephfs2-19.0.0-2568.gafa73a9e.el9.x8 5.5 MB/s | 797 kB 00:00 2024-04-01T13:27:06.356 INFO:teuthology.orchestra.run.smithi087.stdout:(4/32): libradosstriper1-19.0.0-2568.gafa73a9e. 6.4 MB/s | 548 kB 00:00 2024-04-01T13:27:06.545 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-04-01T13:27:06.665 INFO:teuthology.orchestra.run.smithi087.stdout:(5/32): ceph-common-19.0.0-2568.gafa73a9e.el9.x 11 MB/s | 19 MB 00:01 2024-04-01T13:27:06.715 INFO:teuthology.orchestra.run.smithi087.stdout:(6/32): python3-ceph-argparse-19.0.0-2568.gafa7 905 kB/s | 45 kB 00:00 2024-04-01T13:27:06.783 INFO:teuthology.orchestra.run.smithi087.stdout:(7/32): python3-ceph-common-19.0.0-2568.gafa73a 2.0 MB/s | 134 kB 00:00 2024-04-01T13:27:06.875 INFO:teuthology.orchestra.run.smithi087.stdout:(8/32): librgw2-19.0.0-2568.gafa73a9e.el9.x86_6 9.7 MB/s | 5.0 MB 00:00 2024-04-01T13:27:06.900 INFO:teuthology.orchestra.run.smithi087.stdout:(9/32): python3-cephfs-19.0.0-2568.gafa73a9e.el 1.3 MB/s | 159 kB 00:00 2024-04-01T13:27:06.933 INFO:teuthology.orchestra.run.smithi087.stdout:(10/32): python3-rados-19.0.0-2568.gafa73a9e.el 5.3 MB/s | 315 kB 00:00 2024-04-01T13:27:06.959 INFO:teuthology.orchestra.run.smithi087.stdout:(11/32): python3-rbd-19.0.0-2568.gafa73a9e.el9. 4.9 MB/s | 293 kB 00:00 2024-04-01T13:27:06.984 INFO:teuthology.orchestra.run.smithi087.stdout:(12/32): python3-rgw-19.0.0-2568.gafa73a9e.el9. 1.9 MB/s | 99 kB 00:00 2024-04-01T13:27:07.027 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-01T13:27:07.072 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librados2-2:19.0.0-2568.gafa73a9e.el9.x86_64 2/34 2024-04-01T13:27:07.147 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:19.0.0-2568.gafa73a9e.el9.x86_64 2/34 2024-04-01T13:27:07.176 INFO:teuthology.orchestra.run.smithi087.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 488 kB/s | 106 kB 00:00 2024-04-01T13:27:07.218 INFO:teuthology.orchestra.run.smithi087.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 681 kB/s | 159 kB 00:00 2024-04-01T13:27:07.237 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rados-2:19.0.0-2568.gafa73a9e.el9.x86_64 3/34 2024-04-01T13:27:07.265 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs2-2:19.0.0-2568.gafa73a9e.el9.x86_64 4/34 2024-04-01T13:27:07.293 INFO:teuthology.orchestra.run.smithi087.stdout:(15/32): ceph-base-19.0.0-2568.gafa73a9e.el9.x8 2.5 MB/s | 5.9 MB 00:02 2024-04-01T13:27:07.318 INFO:teuthology.orchestra.run.smithi087.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 311 kB/s | 44 kB 00:00 2024-04-01T13:27:07.328 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs2-2:19.0.0-2568.gafa73a9e.el9.x86_64 4/34 2024-04-01T13:27:07.369 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-01T13:27:07.402 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-01T13:27:07.403 INFO:teuthology.orchestra.run.smithi087.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 661 kB 00:00 2024-04-01T13:27:07.427 INFO:teuthology.orchestra.run.smithi087.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 282 kB 00:00 2024-04-01T13:27:07.428 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-01T13:27:07.451 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-01T13:27:07.452 INFO:teuthology.orchestra.run.smithi087.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 308 kB/s | 41 kB 00:00 2024-04-01T13:27:07.467 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-01T13:27:07.478 INFO:teuthology.orchestra.run.smithi087.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2024-04-01T13:27:07.511 INFO:teuthology.orchestra.run.smithi087.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 596 kB/s | 35 kB 00:00 2024-04-01T13:27:07.537 INFO:teuthology.orchestra.run.smithi087.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 555 kB/s | 46 kB 00:00 2024-04-01T13:27:07.614 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-01T13:27:07.649 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-01T13:27:07.837 INFO:teuthology.orchestra.run.smithi087.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 86 kB/s | 26 kB 00:00 2024-04-01T13:27:08.020 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-01T13:27:08.029 INFO:teuthology.orchestra.run.smithi087.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 253 kB/s | 49 kB 00:00 2024-04-01T13:27:08.054 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librbd1-2:19.0.0-2568.gafa73a9e.el9.x86_64 11/34 2024-04-01T13:27:08.054 INFO:teuthology.orchestra.run.smithi087.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 536 kB/s | 309 kB 00:00 2024-04-01T13:27:08.104 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:19.0.0-2568.gafa73a9e.el9.x86_64 11/34 2024-04-01T13:27:08.138 INFO:teuthology.orchestra.run.smithi087.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 620 kB/s | 67 kB 00:00 2024-04-01T13:27:08.140 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-argparse-2:19.0.0-2568.gafa73a9e.el9. 12/34 2024-04-01T13:27:08.195 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cephfs-2:19.0.0-2568.gafa73a9e.el9.x86_64 13/34 2024-04-01T13:27:08.226 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rbd-2:19.0.0-2568.gafa73a9e.el9.x86_64 14/34 2024-04-01T13:27:08.318 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-01T13:27:08.338 INFO:teuthology.orchestra.run.smithi087.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 959 kB/s | 192 kB 00:00 2024-04-01T13:27:08.357 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libradosstriper1-2:19.0.0-2568.gafa73a9e.el9.x86_6 16/34 2024-04-01T13:27:08.372 INFO:teuthology.orchestra.run.smithi087.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 838 kB 00:00 2024-04-01T13:27:08.446 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2568.gafa73a9e.el9.x86_6 16/34 2024-04-01T13:27:08.481 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-01T13:27:08.592 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-01T13:27:08.623 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-01T13:27:08.663 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-01T13:27:08.705 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-01T13:27:08.734 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-01T13:27:08.767 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-01T13:27:08.818 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-01T13:27:08.830 INFO:teuthology.orchestra.run.smithi087.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 3.2 MB/s | 1.6 MB 00:00 2024-04-01T13:27:09.147 INFO:teuthology.orchestra.run.smithi087.stdout:(30/32): librbd1-19.0.0-2568.gafa73a9e.el9.x86_ 12 MB/s | 3.7 MB 00:00 2024-04-01T13:27:09.222 INFO:teuthology.orchestra.run.smithi087.stdout:(31/32): libarrow-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 4.4 MB 00:01 2024-04-01T13:27:09.272 INFO:teuthology.orchestra.run.smithi087.stdout:(32/32): librados2-19.0.0-2568.gafa73a9e.el9.x8 3.8 MB/s | 3.4 MB 00:00 2024-04-01T13:27:09.277 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:27:09.277 INFO:teuthology.orchestra.run.smithi087.stdout:Total 12 MB/s | 58 MB 00:04 2024-04-01T13:27:09.278 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-01T13:27:09.315 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-01T13:27:09.315 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-01T13:27:09.381 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-01T13:27:09.404 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librgw2-2:19.0.0-2568.gafa73a9e.el9.x86_64 26/34 2024-04-01T13:27:09.459 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librgw2-2:19.0.0-2568.gafa73a9e.el9.x86_64 26/34 2024-04-01T13:27:09.529 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rgw-2:19.0.0-2568.gafa73a9e.el9.x86_64 27/34 2024-04-01T13:27:09.565 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-common-2:19.0.0-2568.gafa73a9e.el9.x8 28/34 2024-04-01T13:27:09.791 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-01T13:27:09.791 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-01T13:27:11.293 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-01T13:27:11.777 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-01T13:27:11.808 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librados2-2:19.0.0-2568.gafa73a9e.el9.x86_64 2/34 2024-04-01T13:27:11.899 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:19.0.0-2568.gafa73a9e.el9.x86_64 2/34 2024-04-01T13:27:11.997 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rados-2:19.0.0-2568.gafa73a9e.el9.x86_64 3/34 2024-04-01T13:27:12.026 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs2-2:19.0.0-2568.gafa73a9e.el9.x86_64 4/34 2024-04-01T13:27:12.072 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:19.0.0-2568.gafa73a9e.el9.x86_64 29/34 2024-04-01T13:27:12.089 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs2-2:19.0.0-2568.gafa73a9e.el9.x86_64 4/34 2024-04-01T13:27:12.112 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-common-2:19.0.0-2568.gafa73a9e.el9.x86_64 29/34 2024-04-01T13:27:12.135 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-01T13:27:12.164 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-01T13:27:12.189 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-01T13:27:12.212 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-01T13:27:12.228 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-01T13:27:12.421 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-01T13:27:12.453 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-01T13:27:12.804 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-01T13:27:12.837 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:19.0.0-2568.gafa73a9e.el9.x86_64 29/34 2024-04-01T13:27:12.841 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librbd1-2:19.0.0-2568.gafa73a9e.el9.x86_64 11/34 2024-04-01T13:27:12.873 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-base-2:19.0.0-2568.gafa73a9e.el9.x86_64 30/34 2024-04-01T13:27:12.916 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:19.0.0-2568.gafa73a9e.el9.x86_64 11/34 2024-04-01T13:27:12.945 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-base-2:19.0.0-2568.gafa73a9e.el9.x86_64 30/34 2024-04-01T13:27:12.946 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-01T13:27:12.946 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-01T13:27:12.946 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:27:12.958 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-argparse-2:19.0.0-2568.gafa73a9e.el9. 12/34 2024-04-01T13:27:12.974 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-selinux-2:19.0.0-2568.gafa73a9e.el9.x86_64 31/34 2024-04-01T13:27:13.013 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cephfs-2:19.0.0-2568.gafa73a9e.el9.x86_64 13/34 2024-04-01T13:27:13.038 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rbd-2:19.0.0-2568.gafa73a9e.el9.x86_64 14/34 2024-04-01T13:27:13.116 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-01T13:27:13.144 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libradosstriper1-2:19.0.0-2568.gafa73a9e.el9.x86_6 16/34 2024-04-01T13:27:13.231 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2568.gafa73a9e.el9.x86_6 16/34 2024-04-01T13:27:13.268 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-01T13:27:13.374 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-01T13:27:13.418 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-01T13:27:13.459 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-01T13:27:13.492 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-01T13:27:13.520 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-01T13:27:13.552 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-01T13:27:13.595 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-01T13:27:14.139 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-01T13:27:14.167 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librgw2-2:19.0.0-2568.gafa73a9e.el9.x86_64 26/34 2024-04-01T13:27:14.232 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librgw2-2:19.0.0-2568.gafa73a9e.el9.x86_64 26/34 2024-04-01T13:27:14.306 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rgw-2:19.0.0-2568.gafa73a9e.el9.x86_64 27/34 2024-04-01T13:27:14.336 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-common-2:19.0.0-2568.gafa73a9e.el9.x8 28/34 2024-04-01T13:27:16.727 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:19.0.0-2568.gafa73a9e.el9.x86_64 29/34 2024-04-01T13:27:16.766 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-common-2:19.0.0-2568.gafa73a9e.el9.x86_64 29/34 2024-04-01T13:27:17.439 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:19.0.0-2568.gafa73a9e.el9.x86_64 29/34 2024-04-01T13:27:17.478 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-base-2:19.0.0-2568.gafa73a9e.el9.x86_64 30/34 2024-04-01T13:27:17.580 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-base-2:19.0.0-2568.gafa73a9e.el9.x86_64 30/34 2024-04-01T13:27:17.580 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-01T13:27:17.580 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-01T13:27:17.580 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:27:17.620 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-selinux-2:19.0.0-2568.gafa73a9e.el9.x86_64 31/34 2024-04-01T13:27:23.286 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2568.gafa73a9e.el9.x86_64 31/34 2024-04-01T13:27:23.286 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /sys 2024-04-01T13:27:23.286 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /proc 2024-04-01T13:27:23.286 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /mnt 2024-04-01T13:27:23.286 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /var/tmp 2024-04-01T13:27:23.286 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /home 2024-04-01T13:27:23.287 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /root 2024-04-01T13:27:23.287 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /tmp 2024-04-01T13:27:23.287 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:27:24.319 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-radosgw-2:19.0.0-2568.gafa73a9e.el9.x86_64 32/34 2024-04-01T13:27:24.343 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2568.gafa73a9e.el9.x86_64 32/34 2024-04-01T13:27:24.343 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T13:27:24.343 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-01T13:27:24.344 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-01T13:27:24.344 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-01T13:27:24.344 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:27:24.345 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-01T13:27:24.413 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-01T13:27:24.413 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-01T13:27:25.620 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-01T13:27:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-base-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/34 2024-04-01T13:27:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-common-2:19.0.0-2568.gafa73a9e.el9.x86_64 2/34 2024-04-01T13:27:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-radosgw-2:19.0.0-2568.gafa73a9e.el9.x86_64 3/34 2024-04-01T13:27:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-selinux-2:19.0.0-2568.gafa73a9e.el9.x86_64 4/34 2024-04-01T13:27:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs2-2:19.0.0-2568.gafa73a9e.el9.x86_64 5/34 2024-04-01T13:27:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libradosstriper1-2:19.0.0-2568.gafa73a9e.el9.x86_6 6/34 2024-04-01T13:27:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librgw2-2:19.0.0-2568.gafa73a9e.el9.x86_64 7/34 2024-04-01T13:27:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2568.gafa73a9e.el9. 8/34 2024-04-01T13:27:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-common-2:19.0.0-2568.gafa73a9e.el9.x8 9/34 2024-04-01T13:27:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cephfs-2:19.0.0-2568.gafa73a9e.el9.x86_64 10/34 2024-04-01T13:27:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rados-2:19.0.0-2568.gafa73a9e.el9.x86_64 11/34 2024-04-01T13:27:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rbd-2:19.0.0-2568.gafa73a9e.el9.x86_64 12/34 2024-04-01T13:27:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rgw-2:19.0.0-2568.gafa73a9e.el9.x86_64 13/34 2024-04-01T13:27:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-01T13:27:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-01T13:27:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-01T13:27:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-01T13:27:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-01T13:27:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-01T13:27:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-01T13:27:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-01T13:27:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-01T13:27:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-01T13:27:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-01T13:27:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-01T13:27:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-01T13:27:25.623 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-01T13:27:25.623 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-01T13:27:25.623 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-01T13:27:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-01T13:27:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:19.0.0-2568.gafa73a9e.el9.x86_64 31/34 2024-04-01T13:27:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-01T13:27:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:19.0.0-2568.gafa73a9e.el9.x86_64 33/34 2024-04-01T13:27:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-01T13:27:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:27:26.732 INFO:teuthology.orchestra.run.smithi059.stdout:Upgraded: 2024-04-01T13:27:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: librados2-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:26.732 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-04-01T13:27:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-01T13:27:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-01T13:27:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-01T13:27:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-01T13:27:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-01T13:27:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-01T13:27:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-01T13:27:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-01T13:27:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-01T13:27:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-01T13:27:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-01T13:27:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-01T13:27:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-01T13:27:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-01T13:27:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-01T13:27:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: re2-1:20211101-3.el9.x86_64 2024-04-01T13:27:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-01T13:27:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:27:26.735 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:27:27.000 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-test 2024-04-01T13:27:27.405 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2568.gafa73a9e.el9.x86_64 31/34 2024-04-01T13:27:27.405 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /sys 2024-04-01T13:27:27.406 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /proc 2024-04-01T13:27:27.406 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /mnt 2024-04-01T13:27:27.406 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /var/tmp 2024-04-01T13:27:27.406 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /home 2024-04-01T13:27:27.406 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /root 2024-04-01T13:27:27.406 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /tmp 2024-04-01T13:27:27.406 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:27:27.566 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:32 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:27:27.731 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:27:27.732 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:27:27.732 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-04-01T13:27:27.732 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:27:27.732 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-04-01T13:27:27.733 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 87 M 2024-04-01T13:27:27.733 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-04-01T13:27:27.733 INFO:teuthology.orchestra.run.smithi059.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-01T13:27:27.733 INFO:teuthology.orchestra.run.smithi059.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-01T13:27:27.733 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 173 k 2024-04-01T13:27:27.733 INFO:teuthology.orchestra.run.smithi059.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-01T13:27:27.733 INFO:teuthology.orchestra.run.smithi059.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-04-01T13:27:27.733 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-01T13:27:27.733 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-01T13:27:27.733 INFO:teuthology.orchestra.run.smithi059.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-01T13:27:27.733 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-01T13:27:27.733 INFO:teuthology.orchestra.run.smithi059.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-01T13:27:27.733 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:27:27.733 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-04-01T13:27:27.733 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:27:27.734 INFO:teuthology.orchestra.run.smithi059.stdout:Install 11 Packages 2024-04-01T13:27:27.734 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:27:27.734 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 90 M 2024-04-01T13:27:27.734 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 819 M 2024-04-01T13:27:27.734 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-04-01T13:27:28.317 INFO:teuthology.orchestra.run.smithi059.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-04-01T13:27:28.394 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-radosgw-2:19.0.0-2568.gafa73a9e.el9.x86_64 32/34 2024-04-01T13:27:28.409 INFO:teuthology.orchestra.run.smithi059.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-04-01T13:27:28.418 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2568.gafa73a9e.el9.x86_64 32/34 2024-04-01T13:27:28.418 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T13:27:28.418 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-01T13:27:28.418 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-01T13:27:28.418 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-01T13:27:28.419 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:27:28.420 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-01T13:27:28.425 INFO:teuthology.orchestra.run.smithi059.stdout:(3/11): libcephsqlite-19.0.0-2568.gafa73a9e.el9 584 kB/s | 173 kB 00:00 2024-04-01T13:27:28.459 INFO:teuthology.orchestra.run.smithi059.stdout:(4/11): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-04-01T13:27:28.484 INFO:teuthology.orchestra.run.smithi059.stdout:(5/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-04-01T13:27:28.496 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-01T13:27:28.496 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-01T13:27:28.510 INFO:teuthology.orchestra.run.smithi059.stdout:(6/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 1.8 MB/s | 90 kB 00:00 2024-04-01T13:27:28.727 INFO:teuthology.orchestra.run.smithi059.stdout:(7/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 2.1 MB/s | 469 kB 00:00 2024-04-01T13:27:28.777 INFO:teuthology.orchestra.run.smithi059.stdout:(8/11): libubsan-11.4.1-3.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-04-01T13:27:29.061 INFO:teuthology.orchestra.run.smithi059.stdout:(9/11): jq-1.6-16.el9.x86_64.rpm 325 kB/s | 187 kB 00:00 2024-04-01T13:27:29.619 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-01T13:27:29.619 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-base-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/34 2024-04-01T13:27:29.619 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-common-2:19.0.0-2568.gafa73a9e.el9.x86_64 2/34 2024-04-01T13:27:29.619 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-radosgw-2:19.0.0-2568.gafa73a9e.el9.x86_64 3/34 2024-04-01T13:27:29.619 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-selinux-2:19.0.0-2568.gafa73a9e.el9.x86_64 4/34 2024-04-01T13:27:29.620 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs2-2:19.0.0-2568.gafa73a9e.el9.x86_64 5/34 2024-04-01T13:27:29.620 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libradosstriper1-2:19.0.0-2568.gafa73a9e.el9.x86_6 6/34 2024-04-01T13:27:29.620 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librgw2-2:19.0.0-2568.gafa73a9e.el9.x86_64 7/34 2024-04-01T13:27:29.620 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2568.gafa73a9e.el9. 8/34 2024-04-01T13:27:29.620 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-common-2:19.0.0-2568.gafa73a9e.el9.x8 9/34 2024-04-01T13:27:29.620 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cephfs-2:19.0.0-2568.gafa73a9e.el9.x86_64 10/34 2024-04-01T13:27:29.620 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rados-2:19.0.0-2568.gafa73a9e.el9.x86_64 11/34 2024-04-01T13:27:29.620 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rbd-2:19.0.0-2568.gafa73a9e.el9.x86_64 12/34 2024-04-01T13:27:29.620 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rgw-2:19.0.0-2568.gafa73a9e.el9.x86_64 13/34 2024-04-01T13:27:29.620 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-01T13:27:29.620 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-01T13:27:29.620 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-01T13:27:29.620 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-01T13:27:29.620 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-01T13:27:29.620 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-01T13:27:29.620 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-01T13:27:29.621 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-01T13:27:29.621 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-01T13:27:29.621 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-01T13:27:29.621 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-01T13:27:29.621 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-01T13:27:29.621 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-01T13:27:29.621 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-01T13:27:29.621 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-01T13:27:29.621 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-01T13:27:29.621 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-01T13:27:29.622 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:19.0.0-2568.gafa73a9e.el9.x86_64 31/34 2024-04-01T13:27:29.622 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-01T13:27:29.622 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:19.0.0-2568.gafa73a9e.el9.x86_64 33/34 2024-04-01T13:27:29.686 INFO:teuthology.orchestra.run.smithi059.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 134 kB/s | 122 kB 00:00 2024-04-01T13:27:30.828 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-01T13:27:30.828 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:27:30.828 INFO:teuthology.orchestra.run.smithi087.stdout:Upgraded: 2024-04-01T13:27:30.829 INFO:teuthology.orchestra.run.smithi087.stdout: librados2-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:30.829 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:30.829 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-04-01T13:27:30.829 INFO:teuthology.orchestra.run.smithi087.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-01T13:27:30.829 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:30.829 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:30.829 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:30.829 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:30.829 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-01T13:27:30.829 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-01T13:27:30.829 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-01T13:27:30.830 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:30.830 INFO:teuthology.orchestra.run.smithi087.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-01T13:27:30.830 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-01T13:27:30.830 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-01T13:27:30.830 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:30.830 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-01T13:27:30.830 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:30.830 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-01T13:27:30.830 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-01T13:27:30.830 INFO:teuthology.orchestra.run.smithi087.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-01T13:27:30.830 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:30.831 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:30.831 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:30.831 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-01T13:27:30.831 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:30.831 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:30.831 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:30.831 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-01T13:27:30.831 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-01T13:27:30.831 INFO:teuthology.orchestra.run.smithi087.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-01T13:27:30.831 INFO:teuthology.orchestra.run.smithi087.stdout: re2-1:20211101-3.el9.x86_64 2024-04-01T13:27:30.831 INFO:teuthology.orchestra.run.smithi087.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-01T13:27:30.831 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:27:30.832 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:27:31.125 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-test 2024-04-01T13:27:31.678 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:30 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:27:31.828 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:27:31.829 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:27:31.829 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-04-01T13:27:31.829 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:27:31.829 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-04-01T13:27:31.830 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 87 M 2024-04-01T13:27:31.830 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-04-01T13:27:31.830 INFO:teuthology.orchestra.run.smithi087.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-01T13:27:31.830 INFO:teuthology.orchestra.run.smithi087.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-01T13:27:31.830 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 173 k 2024-04-01T13:27:31.830 INFO:teuthology.orchestra.run.smithi087.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-01T13:27:31.830 INFO:teuthology.orchestra.run.smithi087.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-04-01T13:27:31.830 INFO:teuthology.orchestra.run.smithi087.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-01T13:27:31.830 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-01T13:27:31.830 INFO:teuthology.orchestra.run.smithi087.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-01T13:27:31.830 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-01T13:27:31.831 INFO:teuthology.orchestra.run.smithi087.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-01T13:27:31.831 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:27:31.831 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-01T13:27:31.831 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:27:31.831 INFO:teuthology.orchestra.run.smithi087.stdout:Install 11 Packages 2024-04-01T13:27:31.831 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:27:31.831 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 90 M 2024-04-01T13:27:31.832 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 819 M 2024-04-01T13:27:31.832 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-01T13:27:32.661 INFO:teuthology.orchestra.run.smithi087.stdout:(1/11): libcephsqlite-19.0.0-2568.gafa73a9e.el9 719 kB/s | 173 kB 00:00 2024-04-01T13:27:32.802 INFO:teuthology.orchestra.run.smithi087.stdout:(2/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 566 kB/s | 217 kB 00:00 2024-04-01T13:27:32.928 INFO:teuthology.orchestra.run.smithi087.stdout:(3/11): protobuf-3.14.0-13.el9.x86_64.rpm 3.8 MB/s | 1.0 MB 00:00 2024-04-01T13:27:32.953 INFO:teuthology.orchestra.run.smithi087.stdout:(4/11): socat-1.7.4.1-5.el9.x86_64.rpm 2.0 MB/s | 300 kB 00:00 2024-04-01T13:27:32.995 INFO:teuthology.orchestra.run.smithi087.stdout:(5/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 945 kB/s | 63 kB 00:00 2024-04-01T13:27:33.020 INFO:teuthology.orchestra.run.smithi087.stdout:(6/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 1.3 MB/s | 90 kB 00:00 2024-04-01T13:27:33.654 INFO:teuthology.orchestra.run.smithi087.stdout:(7/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 740 kB/s | 469 kB 00:00 2024-04-01T13:27:33.821 INFO:teuthology.orchestra.run.smithi087.stdout:(8/11): libubsan-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-04-01T13:27:34.029 INFO:teuthology.orchestra.run.smithi087.stdout:(9/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 585 kB/s | 122 kB 00:00 2024-04-01T13:27:34.180 INFO:teuthology.orchestra.run.smithi087.stdout:(10/11): jq-1.6-16.el9.x86_64.rpm 158 kB/s | 187 kB 00:01 2024-04-01T13:27:35.511 INFO:teuthology.orchestra.run.smithi059.stdout:(11/11): ceph-test-19.0.0-2568.gafa73a9e.el9.x8 12 MB/s | 87 MB 00:07 2024-04-01T13:27:35.513 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:27:35.513 INFO:teuthology.orchestra.run.smithi059.stdout:Total 12 MB/s | 90 MB 00:07 2024-04-01T13:27:35.554 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-04-01T13:27:35.580 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-04-01T13:27:35.580 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-04-01T13:27:36.273 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-04-01T13:27:36.274 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-04-01T13:27:37.188 INFO:teuthology.orchestra.run.smithi087.stdout:(11/11): ceph-test-19.0.0-2568.gafa73a9e.el9.x8 18 MB/s | 87 MB 00:04 2024-04-01T13:27:37.192 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:27:37.192 INFO:teuthology.orchestra.run.smithi087.stdout:Total 17 MB/s | 90 MB 00:05 2024-04-01T13:27:37.233 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-01T13:27:37.257 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-01T13:27:37.257 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-01T13:27:37.324 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-04-01T13:27:37.405 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-04-01T13:27:37.445 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-04-01T13:27:37.471 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-01T13:27:37.547 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-01T13:27:37.579 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-04-01T13:27:37.653 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-04-01T13:27:37.702 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-04-01T13:27:37.733 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-04-01T13:27:37.773 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-04-01T13:27:37.817 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-04-01T13:27:37.834 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephsqlite-2:19.0.0-2568.gafa73a9e.el9.x86_64 10/11 2024-04-01T13:27:37.915 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-01T13:27:37.915 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-01T13:27:38.934 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-01T13:27:39.005 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-04-01T13:27:39.044 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-04-01T13:27:39.073 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-01T13:27:39.155 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-01T13:27:39.189 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-04-01T13:27:39.239 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-04-01T13:27:39.287 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-04-01T13:27:39.318 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-04-01T13:27:39.349 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-04-01T13:27:39.393 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-04-01T13:27:39.419 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephsqlite-2:19.0.0-2568.gafa73a9e.el9.x86_64 10/11 2024-04-01T13:27:49.964 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2568.gafa73a9e.el9.x86_64 10/11 2024-04-01T13:27:50.046 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-test-2:19.0.0-2568.gafa73a9e.el9.x86_64 11/11 2024-04-01T13:27:50.881 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2568.gafa73a9e.el9.x86_64 10/11 2024-04-01T13:27:50.960 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-test-2:19.0.0-2568.gafa73a9e.el9.x86_64 11/11 2024-04-01T13:27:56.072 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-test-2:19.0.0-2568.gafa73a9e.el9.x86_64 11/11 2024-04-01T13:27:56.073 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-test-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/11 2024-04-01T13:27:56.073 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephsqlite-2:19.0.0-2568.gafa73a9e.el9.x86_64 2/11 2024-04-01T13:27:56.073 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-04-01T13:27:56.073 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-04-01T13:27:56.073 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-04-01T13:27:56.073 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-04-01T13:27:56.073 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-04-01T13:27:56.073 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-04-01T13:27:56.073 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-04-01T13:27:56.073 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-04-01T13:27:56.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-04-01T13:27:56.504 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:27:56.504 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-04-01T13:27:56.504 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:56.505 INFO:teuthology.orchestra.run.smithi059.stdout: jq-1.6-16.el9.x86_64 2024-04-01T13:27:56.505 INFO:teuthology.orchestra.run.smithi059.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-01T13:27:56.505 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:56.505 INFO:teuthology.orchestra.run.smithi059.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-01T13:27:56.505 INFO:teuthology.orchestra.run.smithi059.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-01T13:27:56.505 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-01T13:27:56.505 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-01T13:27:56.505 INFO:teuthology.orchestra.run.smithi059.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-01T13:27:56.505 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-01T13:27:56.505 INFO:teuthology.orchestra.run.smithi059.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-01T13:27:56.505 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:27:56.505 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:27:56.766 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph 2024-04-01T13:27:56.971 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-test-2:19.0.0-2568.gafa73a9e.el9.x86_64 11/11 2024-04-01T13:27:56.971 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-test-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/11 2024-04-01T13:27:56.971 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephsqlite-2:19.0.0-2568.gafa73a9e.el9.x86_64 2/11 2024-04-01T13:27:56.971 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-04-01T13:27:56.971 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-04-01T13:27:56.971 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-04-01T13:27:56.972 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-04-01T13:27:56.972 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-04-01T13:27:56.972 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-04-01T13:27:56.972 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-04-01T13:27:56.972 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-04-01T13:27:57.341 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:02 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:27:57.419 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-04-01T13:27:57.420 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:27:57.420 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-04-01T13:27:57.420 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:57.420 INFO:teuthology.orchestra.run.smithi087.stdout: jq-1.6-16.el9.x86_64 2024-04-01T13:27:57.420 INFO:teuthology.orchestra.run.smithi087.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-01T13:27:57.420 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:27:57.420 INFO:teuthology.orchestra.run.smithi087.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-01T13:27:57.420 INFO:teuthology.orchestra.run.smithi087.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-01T13:27:57.420 INFO:teuthology.orchestra.run.smithi087.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-01T13:27:57.420 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-01T13:27:57.420 INFO:teuthology.orchestra.run.smithi087.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-01T13:27:57.420 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-01T13:27:57.420 INFO:teuthology.orchestra.run.smithi087.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-01T13:27:57.420 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:27:57.420 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:27:57.499 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:27:57.502 INFO:teuthology.orchestra.run.smithi059.stdout:========================================================================================= 2024-04-01T13:27:57.502 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-04-01T13:27:57.502 INFO:teuthology.orchestra.run.smithi059.stdout:========================================================================================= 2024-04-01T13:27:57.502 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-04-01T13:27:57.502 INFO:teuthology.orchestra.run.smithi059.stdout: ceph x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 6.5 k 2024-04-01T13:27:57.502 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-04-01T13:27:57.502 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 2.2 M 2024-04-01T13:27:57.502 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 1.6 M 2024-04-01T13:27:57.502 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2568.gafa73a9e.el9 ceph-noarch 249 k 2024-04-01T13:27:57.502 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 5.0 M 2024-04-01T13:27:57.502 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 71 M 2024-04-01T13:27:57.503 INFO:teuthology.orchestra.run.smithi059.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-01T13:27:57.503 INFO:teuthology.orchestra.run.smithi059.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-01T13:27:57.503 INFO:teuthology.orchestra.run.smithi059.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-01T13:27:57.503 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-01T13:27:57.503 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-01T13:27:57.503 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-01T13:27:57.503 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-01T13:27:57.503 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-01T13:27:57.503 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-01T13:27:57.504 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-01T13:27:57.504 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-01T13:27:57.504 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-01T13:27:57.504 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-01T13:27:57.504 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-01T13:27:57.504 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-01T13:27:57.504 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-01T13:27:57.504 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-01T13:27:57.504 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-01T13:27:57.504 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-01T13:27:57.504 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-01T13:27:57.504 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-01T13:27:57.504 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-01T13:27:57.504 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-01T13:27:57.504 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-01T13:27:57.504 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-01T13:27:57.505 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-01T13:27:57.505 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-01T13:27:57.505 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-01T13:27:57.505 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-01T13:27:57.505 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-01T13:27:57.505 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:27:57.505 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-04-01T13:27:57.505 INFO:teuthology.orchestra.run.smithi059.stdout:========================================================================================= 2024-04-01T13:27:57.505 INFO:teuthology.orchestra.run.smithi059.stdout:Install 36 Packages 2024-04-01T13:27:57.505 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:27:57.506 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 84 M 2024-04-01T13:27:57.506 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 921 M 2024-04-01T13:27:57.506 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-04-01T13:27:57.733 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph 2024-04-01T13:27:57.952 INFO:teuthology.orchestra.run.smithi059.stdout:(1/36): ceph-19.0.0-2568.gafa73a9e.el9.x86_64.r 40 kB/s | 6.5 kB 00:00 2024-04-01T13:27:58.202 INFO:teuthology.orchestra.run.smithi059.stdout:(2/36): ceph-mgr-19.0.0-2568.gafa73a9e.el9.x86_ 3.8 MB/s | 1.6 MB 00:00 2024-04-01T13:27:58.301 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:57 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:27:58.453 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:27:58.456 INFO:teuthology.orchestra.run.smithi087.stdout:========================================================================================= 2024-04-01T13:27:58.456 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-04-01T13:27:58.456 INFO:teuthology.orchestra.run.smithi087.stdout:========================================================================================= 2024-04-01T13:27:58.456 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-04-01T13:27:58.456 INFO:teuthology.orchestra.run.smithi087.stdout: ceph x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 6.5 k 2024-04-01T13:27:58.456 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-04-01T13:27:58.456 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 2.2 M 2024-04-01T13:27:58.456 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 1.6 M 2024-04-01T13:27:58.456 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2568.gafa73a9e.el9 ceph-noarch 249 k 2024-04-01T13:27:58.456 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 5.0 M 2024-04-01T13:27:58.456 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 71 M 2024-04-01T13:27:58.456 INFO:teuthology.orchestra.run.smithi087.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-01T13:27:58.457 INFO:teuthology.orchestra.run.smithi087.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-01T13:27:58.457 INFO:teuthology.orchestra.run.smithi087.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-01T13:27:58.457 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-01T13:27:58.457 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-01T13:27:58.457 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-01T13:27:58.457 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-01T13:27:58.457 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-01T13:27:58.457 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-01T13:27:58.457 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-01T13:27:58.457 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-01T13:27:58.457 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-01T13:27:58.457 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-01T13:27:58.457 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-01T13:27:58.457 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-01T13:27:58.458 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-01T13:27:58.458 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-01T13:27:58.458 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-01T13:27:58.458 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-01T13:27:58.458 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-01T13:27:58.458 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-01T13:27:58.458 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-01T13:27:58.458 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-01T13:27:58.458 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-01T13:27:58.458 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-01T13:27:58.458 INFO:teuthology.orchestra.run.smithi087.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-01T13:27:58.458 INFO:teuthology.orchestra.run.smithi087.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-01T13:27:58.458 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-01T13:27:58.458 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-01T13:27:58.459 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-01T13:27:58.459 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:27:58.459 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-01T13:27:58.459 INFO:teuthology.orchestra.run.smithi087.stdout:========================================================================================= 2024-04-01T13:27:58.459 INFO:teuthology.orchestra.run.smithi087.stdout:Install 36 Packages 2024-04-01T13:27:58.459 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:27:58.460 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 84 M 2024-04-01T13:27:58.460 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 921 M 2024-04-01T13:27:58.460 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-01T13:27:58.585 INFO:teuthology.orchestra.run.smithi059.stdout:(3/36): ceph-mon-19.0.0-2568.gafa73a9e.el9.x86_ 7.9 MB/s | 5.0 MB 00:00 2024-04-01T13:27:58.627 INFO:teuthology.orchestra.run.smithi059.stdout:(4/36): ceph-mds-19.0.0-2568.gafa73a9e.el9.x86_ 2.6 MB/s | 2.2 MB 00:00 2024-04-01T13:27:58.669 INFO:teuthology.orchestra.run.smithi059.stdout:(5/36): ceph-mgr-modules-core-19.0.0-2568.gafa7 2.9 MB/s | 249 kB 00:00 2024-04-01T13:27:58.795 INFO:teuthology.orchestra.run.smithi059.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-04-01T13:27:58.820 INFO:teuthology.orchestra.run.smithi059.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 961 kB/s | 144 kB 00:00 2024-04-01T13:27:58.845 INFO:teuthology.orchestra.run.smithi059.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 629 kB/s | 32 kB 00:00 2024-04-01T13:27:58.870 INFO:teuthology.orchestra.run.smithi059.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 944 kB/s | 47 kB 00:00 2024-04-01T13:27:58.895 INFO:teuthology.orchestra.run.smithi059.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 774 kB/s | 39 kB 00:00 2024-04-01T13:27:58.921 INFO:teuthology.orchestra.run.smithi059.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-04-01T13:27:58.980 INFO:teuthology.orchestra.run.smithi059.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-04-01T13:27:59.005 INFO:teuthology.orchestra.run.smithi059.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-04-01T13:27:59.030 INFO:teuthology.orchestra.run.smithi059.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-04-01T13:27:59.055 INFO:teuthology.orchestra.run.smithi059.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-04-01T13:27:59.139 INFO:teuthology.orchestra.run.smithi059.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-04-01T13:27:59.193 INFO:teuthology.orchestra.run.smithi087.stdout:(1/36): ceph-19.0.0-2568.gafa73a9e.el9.x86_64.r 21 kB/s | 6.5 kB 00:00 2024-04-01T13:27:59.289 INFO:teuthology.orchestra.run.smithi059.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 482 kB/s | 125 kB 00:00 2024-04-01T13:27:59.373 INFO:teuthology.orchestra.run.smithi059.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 724 kB/s | 151 kB 00:00 2024-04-01T13:27:59.398 INFO:teuthology.orchestra.run.smithi059.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 401 kB/s | 43 kB 00:00 2024-04-01T13:27:59.423 INFO:teuthology.orchestra.run.smithi059.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-04-01T13:27:59.474 INFO:teuthology.orchestra.run.smithi059.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.7 MB/s | 358 kB 00:00 2024-04-01T13:27:59.499 INFO:teuthology.orchestra.run.smithi059.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 143 kB/s | 11 kB 00:00 2024-04-01T13:27:59.524 INFO:teuthology.orchestra.run.smithi059.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-04-01T13:27:59.549 INFO:teuthology.orchestra.run.smithi059.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 464 kB/s | 23 kB 00:00 2024-04-01T13:27:59.575 INFO:teuthology.orchestra.run.smithi059.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-04-01T13:27:59.600 INFO:teuthology.orchestra.run.smithi059.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-04-01T13:27:59.625 INFO:teuthology.orchestra.run.smithi059.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-04-01T13:27:59.643 INFO:teuthology.orchestra.run.smithi087.stdout:(2/36): ceph-mgr-19.0.0-2568.gafa73a9e.el9.x86_ 2.1 MB/s | 1.6 MB 00:00 2024-04-01T13:27:59.650 INFO:teuthology.orchestra.run.smithi059.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-04-01T13:27:59.676 INFO:teuthology.orchestra.run.smithi059.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-01T13:27:59.701 INFO:teuthology.orchestra.run.smithi059.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-04-01T13:27:59.718 INFO:teuthology.orchestra.run.smithi059.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-04-01T13:27:59.751 INFO:teuthology.orchestra.run.smithi059.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-04-01T13:27:59.777 INFO:teuthology.orchestra.run.smithi059.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-04-01T13:27:59.811 INFO:teuthology.orchestra.run.smithi059.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.2 MB/s | 427 kB 00:00 2024-04-01T13:27:59.836 INFO:teuthology.orchestra.run.smithi059.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-04-01T13:28:02.101 INFO:teuthology.orchestra.run.smithi087.stdout:(3/36): ceph-mon-19.0.0-2568.gafa73a9e.el9.x86_ 1.7 MB/s | 5.0 MB 00:02 2024-04-01T13:28:02.210 INFO:teuthology.orchestra.run.smithi087.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2568.gafa7 2.2 MB/s | 249 kB 00:00 2024-04-01T13:28:02.377 INFO:teuthology.orchestra.run.smithi087.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-04-01T13:28:02.386 INFO:teuthology.orchestra.run.smithi059.stdout:(36/36): ceph-osd-19.0.0-2568.gafa73a9e.el9.x86 17 MB/s | 71 MB 00:04 2024-04-01T13:28:02.392 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:28:02.393 INFO:teuthology.orchestra.run.smithi059.stdout:Total 17 MB/s | 84 MB 00:04 2024-04-01T13:28:02.418 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-04-01T13:28:02.427 INFO:teuthology.orchestra.run.smithi087.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-04-01T13:28:02.466 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-04-01T13:28:02.467 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-04-01T13:28:02.477 INFO:teuthology.orchestra.run.smithi087.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 630 kB/s | 32 kB 00:00 2024-04-01T13:28:02.528 INFO:teuthology.orchestra.run.smithi087.stdout:(8/36): ceph-mds-19.0.0-2568.gafa73a9e.el9.x86_ 614 kB/s | 2.2 MB 00:03 2024-04-01T13:28:02.553 INFO:teuthology.orchestra.run.smithi087.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 629 kB/s | 47 kB 00:00 2024-04-01T13:28:02.587 INFO:teuthology.orchestra.run.smithi087.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 662 kB/s | 39 kB 00:00 2024-04-01T13:28:02.612 INFO:teuthology.orchestra.run.smithi087.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-04-01T13:28:02.704 INFO:teuthology.orchestra.run.smithi087.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 9.8 MB/s | 1.1 MB 00:00 2024-04-01T13:28:02.729 INFO:teuthology.orchestra.run.smithi087.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 879 kB/s | 103 kB 00:00 2024-04-01T13:28:02.754 INFO:teuthology.orchestra.run.smithi087.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-04-01T13:28:02.788 INFO:teuthology.orchestra.run.smithi087.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-04-01T13:28:02.972 INFO:teuthology.orchestra.run.smithi087.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 575 kB/s | 125 kB 00:00 2024-04-01T13:28:03.226 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-04-01T13:28:03.227 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-04-01T13:28:03.522 INFO:teuthology.orchestra.run.smithi087.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 30 kB/s | 22 kB 00:00 2024-04-01T13:28:03.972 INFO:teuthology.orchestra.run.smithi087.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 151 kB/s | 151 kB 00:00 2024-04-01T13:28:04.947 INFO:teuthology.orchestra.run.smithi087.stdout:(19/36): ceph-osd-19.0.0-2568.gafa73a9e.el9.x86 13 MB/s | 71 MB 00:05 2024-04-01T13:28:04.970 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-04-01T13:28:04.972 INFO:teuthology.orchestra.run.smithi087.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 30 kB/s | 43 kB 00:01 2024-04-01T13:28:04.998 INFO:teuthology.orchestra.run.smithi087.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 167 kB/s | 172 kB 00:01 2024-04-01T13:28:05.022 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-01T13:28:05.064 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-01T13:28:05.091 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-01T13:28:05.171 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-01T13:28:05.224 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-01T13:28:05.282 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-01T13:28:05.385 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-01T13:28:05.466 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-01T13:28:05.519 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-01T13:28:05.596 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-01T13:28:05.655 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-01T13:28:05.699 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-01T13:28:05.770 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-01T13:28:05.840 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-01T13:28:06.053 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-01T13:28:06.103 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-01T13:28:06.141 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-01T13:28:06.175 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-01T13:28:06.205 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-01T13:28:06.288 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-01T13:28:06.324 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-01T13:28:06.357 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-01T13:28:06.401 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-01T13:28:06.442 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-01T13:28:06.456 INFO:teuthology.orchestra.run.smithi087.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 238 kB/s | 358 kB 00:01 2024-04-01T13:28:06.475 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-01T13:28:06.567 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-01T13:28:06.606 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-01T13:28:06.686 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-01T13:28:06.842 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-01T13:28:06.990 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-01T13:28:07.185 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2568.gafa73a9e.el9. 31/36 2024-04-01T13:28:07.209 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-2:19.0.0-2568.gafa73a9e.el9.x86_64 32/36 2024-04-01T13:28:07.225 INFO:teuthology.orchestra.run.smithi087.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 30 kB/s | 23 kB 00:00 2024-04-01T13:28:07.235 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2568.gafa73a9e.el9.x86_64 32/36 2024-04-01T13:28:07.235 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T13:28:07.235 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-01T13:28:07.235 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-01T13:28:07.235 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-01T13:28:07.235 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:07.573 INFO:teuthology.orchestra.run.smithi087.stdout:(24/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 4.1 kB/s | 11 kB 00:02 2024-04-01T13:28:07.590 INFO:teuthology.orchestra.run.smithi087.stdout:(25/36): python3-jaraco-classes-3.2.1-5.el9.noa 6.8 kB/s | 18 kB 00:02 2024-04-01T13:28:08.307 INFO:teuthology.orchestra.run.smithi087.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 18 kB/s | 19 kB 00:01 2024-04-01T13:28:10.332 INFO:teuthology.orchestra.run.smithi087.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 7.1 kB/s | 20 kB 00:02 2024-04-01T13:28:10.349 INFO:teuthology.orchestra.run.smithi087.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 17 kB/s | 46 kB 00:02 2024-04-01T13:28:10.374 INFO:teuthology.orchestra.run.smithi087.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 38 kB/s | 79 kB 00:02 2024-04-01T13:28:10.741 INFO:teuthology.orchestra.run.smithi087.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 245 kB/s | 90 kB 00:00 2024-04-01T13:28:10.850 INFO:teuthology.orchestra.run.smithi087.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 526 kB/s | 272 kB 00:00 2024-04-01T13:28:11.058 INFO:teuthology.orchestra.run.smithi087.stdout:(32/36): python3-portend-3.1.0-2.el9.noarch.rpm 23 kB/s | 16 kB 00:00 2024-04-01T13:28:13.775 INFO:teuthology.orchestra.run.smithi087.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 12 kB/s | 36 kB 00:03 2024-04-01T13:28:13.800 INFO:teuthology.orchestra.run.smithi087.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 78 kB/s | 230 kB 00:02 2024-04-01T13:28:13.975 INFO:teuthology.orchestra.run.smithi087.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 146 kB/s | 427 kB 00:02 2024-04-01T13:28:14.309 INFO:teuthology.orchestra.run.smithi087.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 37 kB/s | 20 kB 00:00 2024-04-01T13:28:14.312 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:28:14.313 INFO:teuthology.orchestra.run.smithi087.stdout:Total 5.3 MB/s | 84 MB 00:15 2024-04-01T13:28:14.337 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-01T13:28:14.381 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-01T13:28:14.381 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-01T13:28:15.114 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-01T13:28:15.114 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-01T13:28:16.706 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-01T13:28:16.778 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-01T13:28:16.811 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-01T13:28:16.839 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-01T13:28:16.909 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-01T13:28:16.954 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-01T13:28:16.987 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-01T13:28:17.078 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-01T13:28:17.156 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-01T13:28:17.206 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-01T13:28:17.278 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-01T13:28:17.331 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-01T13:28:17.369 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-01T13:28:17.419 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-01T13:28:17.474 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-01T13:28:17.660 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-01T13:28:17.716 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-01T13:28:17.754 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-01T13:28:17.788 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-01T13:28:17.818 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-01T13:28:17.895 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-01T13:28:17.929 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-01T13:28:17.962 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-01T13:28:18.002 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-01T13:28:18.037 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-01T13:28:18.080 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-01T13:28:18.168 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-01T13:28:18.203 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-01T13:28:18.274 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-01T13:28:18.410 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-01T13:28:18.526 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-01T13:28:18.716 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2568.gafa73a9e.el9. 31/36 2024-04-01T13:28:18.740 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-2:19.0.0-2568.gafa73a9e.el9.x86_64 32/36 2024-04-01T13:28:18.751 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-osd-2:19.0.0-2568.gafa73a9e.el9.x86_64 33/36 2024-04-01T13:28:18.766 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2568.gafa73a9e.el9.x86_64 32/36 2024-04-01T13:28:18.766 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T13:28:18.766 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-01T13:28:18.766 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-01T13:28:18.766 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-01T13:28:18.766 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:18.777 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-osd-2:19.0.0-2568.gafa73a9e.el9.x86_64 33/36 2024-04-01T13:28:18.777 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T13:28:18.777 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-01T13:28:18.778 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-01T13:28:18.778 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-01T13:28:18.778 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:19.343 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mon-2:19.0.0-2568.gafa73a9e.el9.x86_64 34/36 2024-04-01T13:28:19.366 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mon-2:19.0.0-2568.gafa73a9e.el9.x86_64 34/36 2024-04-01T13:28:19.367 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T13:28:19.367 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-01T13:28:19.367 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-01T13:28:19.367 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-01T13:28:19.367 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:19.611 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mds-2:19.0.0-2568.gafa73a9e.el9.x86_64 35/36 2024-04-01T13:28:19.634 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mds-2:19.0.0-2568.gafa73a9e.el9.x86_64 35/36 2024-04-01T13:28:19.634 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T13:28:19.634 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-01T13:28:19.634 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-01T13:28:19.634 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-01T13:28:19.634 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:19.702 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-2:19.0.0-2568.gafa73a9e.el9.x86_64 36/36 2024-04-01T13:28:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-2:19.0.0-2568.gafa73a9e.el9.x86_64 36/36 2024-04-01T13:28:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/36 2024-04-01T13:28:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mds-2:19.0.0-2568.gafa73a9e.el9.x86_64 2/36 2024-04-01T13:28:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-2:19.0.0-2568.gafa73a9e.el9.x86_64 3/36 2024-04-01T13:28:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mon-2:19.0.0-2568.gafa73a9e.el9.x86_64 4/36 2024-04-01T13:28:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-osd-2:19.0.0-2568.gafa73a9e.el9.x86_64 5/36 2024-04-01T13:28:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2568.gafa73a9e.el9. 6/36 2024-04-01T13:28:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-01T13:28:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-01T13:28:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-01T13:28:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-01T13:28:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-01T13:28:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-01T13:28:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-01T13:28:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-01T13:28:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-01T13:28:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-01T13:28:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-01T13:28:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-01T13:28:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-01T13:28:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-01T13:28:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-01T13:28:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-01T13:28:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-01T13:28:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-01T13:28:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-01T13:28:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-01T13:28:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-01T13:28:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-01T13:28:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-01T13:28:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-01T13:28:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-01T13:28:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-01T13:28:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-01T13:28:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-01T13:28:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-01T13:28:27.272 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-01T13:28:27.272 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:27.272 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-04-01T13:28:27.272 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:28:27.272 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:28:27.272 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:28:27.272 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core-2:19.0.0-2568.gafa73a9e.el9.noarch 2024-04-01T13:28:27.272 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:28:27.272 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:28:27.272 INFO:teuthology.orchestra.run.smithi059.stdout: lua-5.4.4-4.el9.x86_64 2024-04-01T13:28:27.273 INFO:teuthology.orchestra.run.smithi059.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-01T13:28:27.273 INFO:teuthology.orchestra.run.smithi059.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-01T13:28:27.273 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-01T13:28:27.273 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-01T13:28:27.273 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-01T13:28:27.273 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-01T13:28:27.273 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-01T13:28:27.273 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-01T13:28:27.273 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-01T13:28:27.273 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-01T13:28:27.273 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-01T13:28:27.273 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-01T13:28:27.273 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-01T13:28:27.273 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-01T13:28:27.273 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-01T13:28:27.274 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-01T13:28:27.274 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-01T13:28:27.274 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply-3.11-14.el9.noarch 2024-04-01T13:28:27.274 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-01T13:28:27.274 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-01T13:28:27.274 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-01T13:28:27.274 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-01T13:28:27.274 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-01T13:28:27.274 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-01T13:28:27.274 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-01T13:28:27.274 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-01T13:28:27.274 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-01T13:28:27.274 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-01T13:28:27.275 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-01T13:28:27.275 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:27.275 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:28:27.497 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-base 2024-04-01T13:28:28.082 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:33 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:28:28.187 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-base-2:19.0.0-2568.gafa73a9e.el9.x86_64 is already installed. 2024-04-01T13:28:28.243 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:28:28.244 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-04-01T13:28:28.244 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:28:28.304 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install cephadm 2024-04-01T13:28:28.889 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:33 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:28:29.044 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:28:29.045 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:28:29.045 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-04-01T13:28:29.045 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:28:29.045 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-04-01T13:28:29.045 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm noarch 2:19.0.0-2568.gafa73a9e.el9 ceph-noarch 762 k 2024-04-01T13:28:29.045 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:29.045 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-04-01T13:28:29.045 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:28:29.046 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-04-01T13:28:29.046 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:29.046 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 762 k 2024-04-01T13:28:29.046 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 768 k 2024-04-01T13:28:29.046 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-04-01T13:28:29.541 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-osd-2:19.0.0-2568.gafa73a9e.el9.x86_64 33/36 2024-04-01T13:28:29.568 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-osd-2:19.0.0-2568.gafa73a9e.el9.x86_64 33/36 2024-04-01T13:28:29.568 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T13:28:29.568 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-01T13:28:29.568 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-01T13:28:29.568 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-01T13:28:29.568 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:29.706 INFO:teuthology.orchestra.run.smithi059.stdout:cephadm-19.0.0-2568.gafa73a9e.el9.noarch.rpm 1.1 MB/s | 762 kB 00:00 2024-04-01T13:28:29.707 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:28:29.707 INFO:teuthology.orchestra.run.smithi059.stdout:Total 1.1 MB/s | 762 kB 00:00 2024-04-01T13:28:29.707 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-04-01T13:28:29.713 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-04-01T13:28:29.713 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-04-01T13:28:29.719 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-04-01T13:28:29.719 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-04-01T13:28:29.796 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-04-01T13:28:30.109 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mon-2:19.0.0-2568.gafa73a9e.el9.x86_64 34/36 2024-04-01T13:28:30.131 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mon-2:19.0.0-2568.gafa73a9e.el9.x86_64 34/36 2024-04-01T13:28:30.131 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T13:28:30.131 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-01T13:28:30.132 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-01T13:28:30.132 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-01T13:28:30.132 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:30.234 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:19.0.0-2568.gafa73a9e.el9.noarch 1/1 2024-04-01T13:28:30.330 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephadm-2:19.0.0-2568.gafa73a9e.el9.noarch 1/1 2024-04-01T13:28:30.359 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mds-2:19.0.0-2568.gafa73a9e.el9.x86_64 35/36 2024-04-01T13:28:30.382 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mds-2:19.0.0-2568.gafa73a9e.el9.x86_64 35/36 2024-04-01T13:28:30.382 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T13:28:30.382 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-01T13:28:30.382 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-01T13:28:30.382 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-01T13:28:30.382 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:30.447 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-2:19.0.0-2568.gafa73a9e.el9.x86_64 36/36 2024-04-01T13:28:30.719 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:19.0.0-2568.gafa73a9e.el9.noarch 1/1 2024-04-01T13:28:30.983 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephadm-2:19.0.0-2568.gafa73a9e.el9.noarch 1/1 2024-04-01T13:28:30.984 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:30.984 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-04-01T13:28:30.984 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm-2:19.0.0-2568.gafa73a9e.el9.noarch 2024-04-01T13:28:30.984 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:30.984 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:28:31.173 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-immutable-object-cache 2024-04-01T13:28:31.746 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:36 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:28:31.904 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:28:31.904 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:28:31.904 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-04-01T13:28:31.904 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:28:31.904 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-04-01T13:28:31.905 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 157 k 2024-04-01T13:28:31.905 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:31.905 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-04-01T13:28:31.905 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:28:31.905 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-04-01T13:28:31.905 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:31.905 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 157 k 2024-04-01T13:28:31.905 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 640 k 2024-04-01T13:28:31.906 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-04-01T13:28:32.254 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-immutable-object-cache-19.0.0-2568.gafa73a 451 kB/s | 157 kB 00:00 2024-04-01T13:28:32.254 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:28:32.254 INFO:teuthology.orchestra.run.smithi059.stdout:Total 450 kB/s | 157 kB 00:00 2024-04-01T13:28:32.254 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-04-01T13:28:32.264 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-04-01T13:28:32.265 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-04-01T13:28:32.319 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-04-01T13:28:32.319 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-04-01T13:28:32.476 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-04-01T13:28:32.529 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2568.gafa73a9e. 1/1 2024-04-01T13:28:32.553 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2568.gafa73a9e. 1/1 2024-04-01T13:28:32.553 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T13:28:32.553 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-01T13:28:32.553 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:33.565 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2568.gafa73a9e. 1/1 2024-04-01T13:28:33.565 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:33.565 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-04-01T13:28:33.565 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:28:33.565 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:33.565 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:28:33.790 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr 2024-04-01T13:28:34.366 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:39 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:28:34.467 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-mgr-2:19.0.0-2568.gafa73a9e.el9.x86_64 is already installed. 2024-04-01T13:28:34.524 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:28:34.526 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-04-01T13:28:34.526 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:28:34.585 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-dashboard 2024-04-01T13:28:35.161 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:40 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:28:35.319 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:28:35.320 INFO:teuthology.orchestra.run.smithi059.stdout:====================================================================================== 2024-04-01T13:28:35.320 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-04-01T13:28:35.320 INFO:teuthology.orchestra.run.smithi059.stdout:====================================================================================== 2024-04-01T13:28:35.320 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-04-01T13:28:35.320 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2568.gafa73a9e.el9 ceph-noarch 3.5 M 2024-04-01T13:28:35.320 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-04-01T13:28:35.320 INFO:teuthology.orchestra.run.smithi059.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-01T13:28:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2568.gafa73a9e.el9 ceph-noarch 34 k 2024-04-01T13:28:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2568.gafa73a9e.el9 ceph-noarch 16 k 2024-04-01T13:28:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-01T13:28:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-01T13:28:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-01T13:28:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-01T13:28:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-01T13:28:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-01T13:28:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-01T13:28:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:35.321 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-04-01T13:28:35.321 INFO:teuthology.orchestra.run.smithi059.stdout:====================================================================================== 2024-04-01T13:28:35.321 INFO:teuthology.orchestra.run.smithi059.stdout:Install 11 Packages 2024-04-01T13:28:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:35.322 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 7.6 M 2024-04-01T13:28:35.322 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 100 M 2024-04-01T13:28:35.322 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-04-01T13:28:35.998 INFO:teuthology.orchestra.run.smithi059.stdout:(1/11): ceph-grafana-dashboards-19.0.0-2568.gaf 132 kB/s | 34 kB 00:00 2024-04-01T13:28:36.015 INFO:teuthology.orchestra.run.smithi059.stdout:(2/11): ceph-prometheus-alerts-19.0.0-2568.gafa 60 kB/s | 16 kB 00:00 2024-04-01T13:28:36.149 INFO:teuthology.orchestra.run.smithi059.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-04-01T13:28:36.407 INFO:teuthology.orchestra.run.smithi059.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.1 MB/s | 548 kB 00:00 2024-04-01T13:28:36.466 INFO:teuthology.orchestra.run.smithi059.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 333 kB/s | 19 kB 00:00 2024-04-01T13:28:36.783 INFO:teuthology.orchestra.run.smithi059.stdout:(6/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 6.4 MB/s | 2.0 MB 00:00 2024-04-01T13:28:36.850 INFO:teuthology.orchestra.run.smithi059.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-04-01T13:28:36.863 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-2:19.0.0-2568.gafa73a9e.el9.x86_64 36/36 2024-04-01T13:28:36.863 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/36 2024-04-01T13:28:36.863 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mds-2:19.0.0-2568.gafa73a9e.el9.x86_64 2/36 2024-04-01T13:28:36.864 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-2:19.0.0-2568.gafa73a9e.el9.x86_64 3/36 2024-04-01T13:28:36.864 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mon-2:19.0.0-2568.gafa73a9e.el9.x86_64 4/36 2024-04-01T13:28:36.864 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-osd-2:19.0.0-2568.gafa73a9e.el9.x86_64 5/36 2024-04-01T13:28:36.864 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2568.gafa73a9e.el9. 6/36 2024-04-01T13:28:36.864 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-01T13:28:36.864 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-01T13:28:36.864 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-01T13:28:36.864 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-01T13:28:36.864 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-01T13:28:36.864 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-01T13:28:36.864 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-01T13:28:36.864 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-01T13:28:36.864 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-01T13:28:36.864 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-01T13:28:36.865 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-01T13:28:36.865 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-01T13:28:36.865 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-01T13:28:36.865 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-01T13:28:36.865 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-01T13:28:36.865 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-01T13:28:36.865 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-01T13:28:36.865 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-01T13:28:36.865 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-01T13:28:36.865 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-01T13:28:36.865 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-01T13:28:36.865 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-01T13:28:36.865 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-01T13:28:36.865 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-01T13:28:36.866 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-01T13:28:36.866 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-01T13:28:36.866 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-01T13:28:36.866 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-01T13:28:36.866 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-01T13:28:36.908 INFO:teuthology.orchestra.run.smithi059.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 527 kB/s | 31 kB 00:00 2024-04-01T13:28:36.975 INFO:teuthology.orchestra.run.smithi059.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-04-01T13:28:37.150 INFO:teuthology.orchestra.run.smithi059.stdout:(10/11): ceph-mgr-dashboard-19.0.0-2568.gafa73a 2.5 MB/s | 3.5 MB 00:01 2024-04-01T13:28:37.732 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-01T13:28:37.732 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:37.732 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-04-01T13:28:37.732 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:28:37.733 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:28:37.733 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:28:37.733 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core-2:19.0.0-2568.gafa73a9e.el9.noarch 2024-04-01T13:28:37.733 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:28:37.733 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:28:37.733 INFO:teuthology.orchestra.run.smithi087.stdout: lua-5.4.4-4.el9.x86_64 2024-04-01T13:28:37.733 INFO:teuthology.orchestra.run.smithi087.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-01T13:28:37.733 INFO:teuthology.orchestra.run.smithi087.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-01T13:28:37.733 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-01T13:28:37.733 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-01T13:28:37.733 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-01T13:28:37.733 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-01T13:28:37.733 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-01T13:28:37.733 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-01T13:28:37.734 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-01T13:28:37.734 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-01T13:28:37.734 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-01T13:28:37.734 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-01T13:28:37.734 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-01T13:28:37.734 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-01T13:28:37.734 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-01T13:28:37.734 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-01T13:28:37.734 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-01T13:28:37.734 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ply-3.11-14.el9.noarch 2024-04-01T13:28:37.734 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-01T13:28:37.734 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-01T13:28:37.734 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-01T13:28:37.734 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-01T13:28:37.735 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-01T13:28:37.735 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-01T13:28:37.735 INFO:teuthology.orchestra.run.smithi087.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-01T13:28:37.735 INFO:teuthology.orchestra.run.smithi087.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-01T13:28:37.735 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-01T13:28:37.735 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-01T13:28:37.735 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-01T13:28:37.735 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:37.735 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:28:37.801 INFO:teuthology.orchestra.run.smithi059.stdout:(11/11): protobuf-compiler-3.14.0-13.el9.x86_64 490 kB/s | 863 kB 00:01 2024-04-01T13:28:37.805 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:28:37.805 INFO:teuthology.orchestra.run.smithi059.stdout:Total 3.1 MB/s | 7.6 MB 00:02 2024-04-01T13:28:37.826 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-04-01T13:28:37.846 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-04-01T13:28:37.846 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-04-01T13:28:37.967 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-base 2024-04-01T13:28:38.043 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-04-01T13:28:38.044 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-04-01T13:28:38.525 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-04-01T13:28:38.529 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:37 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:28:38.617 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-01T13:28:38.626 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-base-2:19.0.0-2568.gafa73a9e.el9.x86_64 is already installed. 2024-04-01T13:28:38.646 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-01T13:28:38.679 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:28:38.680 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-04-01T13:28:38.680 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:28:38.746 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install cephadm 2024-04-01T13:28:38.747 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-01T13:28:38.886 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-01T13:28:38.932 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-01T13:28:39.027 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-01T13:28:39.076 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-01T13:28:39.104 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-01T13:28:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2568.gafa73a9e.el9 9/11 2024-04-01T13:28:39.308 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:38 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:28:39.459 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:28:39.459 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:28:39.459 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-04-01T13:28:39.459 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:28:39.460 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-04-01T13:28:39.460 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm noarch 2:19.0.0-2568.gafa73a9e.el9 ceph-noarch 762 k 2024-04-01T13:28:39.460 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:39.460 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-01T13:28:39.460 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:28:39.460 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-04-01T13:28:39.460 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:39.460 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 762 k 2024-04-01T13:28:39.460 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 768 k 2024-04-01T13:28:39.460 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-01T13:28:40.250 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2568.gafa73a9e.el 10/11 2024-04-01T13:28:40.287 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2568.gafa73a9e.el9.noa 11/11 2024-04-01T13:28:40.495 INFO:teuthology.orchestra.run.smithi087.stdout:cephadm-19.0.0-2568.gafa73a9e.el9.noarch.rpm 736 kB/s | 762 kB 00:01 2024-04-01T13:28:40.496 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:28:40.496 INFO:teuthology.orchestra.run.smithi087.stdout:Total 735 kB/s | 762 kB 00:01 2024-04-01T13:28:40.496 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-01T13:28:40.502 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-01T13:28:40.502 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-01T13:28:40.507 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-01T13:28:40.508 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-01T13:28:40.585 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-01T13:28:41.029 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:19.0.0-2568.gafa73a9e.el9.noarch 1/1 2024-04-01T13:28:41.125 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : cephadm-2:19.0.0-2568.gafa73a9e.el9.noarch 1/1 2024-04-01T13:28:41.504 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:19.0.0-2568.gafa73a9e.el9.noarch 1/1 2024-04-01T13:28:41.555 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2568.gafa73a9e.el9.noa 11/11 2024-04-01T13:28:41.555 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2568.gafa73a9e.el 1/11 2024-04-01T13:28:41.555 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2568.gafa73a9e.el9.noa 2/11 2024-04-01T13:28:41.555 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2568.gafa73a9e.el9 3/11 2024-04-01T13:28:41.556 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-01T13:28:41.556 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-01T13:28:41.556 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-01T13:28:41.556 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-01T13:28:41.556 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-01T13:28:41.556 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-01T13:28:41.556 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-01T13:28:41.789 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : cephadm-2:19.0.0-2568.gafa73a9e.el9.noarch 1/1 2024-04-01T13:28:41.789 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:41.789 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-04-01T13:28:41.789 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm-2:19.0.0-2568.gafa73a9e.el9.noarch 2024-04-01T13:28:41.789 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:41.789 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:28:41.891 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-01T13:28:41.891 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:41.891 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-04-01T13:28:41.891 INFO:teuthology.orchestra.run.smithi059.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-01T13:28:41.891 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards-2:19.0.0-2568.gafa73a9e.el9.noarch 2024-04-01T13:28:41.891 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard-2:19.0.0-2568.gafa73a9e.el9.noarch 2024-04-01T13:28:41.892 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts-2:19.0.0-2568.gafa73a9e.el9.noarch 2024-04-01T13:28:41.892 INFO:teuthology.orchestra.run.smithi059.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-01T13:28:41.892 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-01T13:28:41.892 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-01T13:28:41.892 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-01T13:28:41.892 INFO:teuthology.orchestra.run.smithi059.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-01T13:28:41.892 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-01T13:28:41.892 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-01T13:28:41.892 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:41.892 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:28:41.961 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-immutable-object-cache 2024-04-01T13:28:42.123 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-01T13:28:42.526 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:41 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:28:42.680 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:28:42.680 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:28:42.681 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-04-01T13:28:42.681 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:28:42.681 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-04-01T13:28:42.681 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 157 k 2024-04-01T13:28:42.681 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:42.681 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-01T13:28:42.681 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:28:42.681 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-04-01T13:28:42.681 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:42.681 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 157 k 2024-04-01T13:28:42.681 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 640 k 2024-04-01T13:28:42.681 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-01T13:28:42.693 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:47 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:28:42.853 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:28:42.854 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================================ 2024-04-01T13:28:42.854 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-04-01T13:28:42.854 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================================ 2024-04-01T13:28:42.854 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-04-01T13:28:42.854 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2568.gafa73a9e.el9 ceph-noarch 7.4 M 2024-04-01T13:28:42.854 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-04-01T13:28:42.854 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-01T13:28:42.855 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-01T13:28:42.855 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-01T13:28:42.855 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-01T13:28:42.855 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-01T13:28:42.855 INFO:teuthology.orchestra.run.smithi059.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-01T13:28:42.855 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-01T13:28:42.855 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-01T13:28:42.855 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-01T13:28:42.855 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-01T13:28:42.855 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-01T13:28:42.855 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:42.855 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-04-01T13:28:42.855 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================================ 2024-04-01T13:28:42.855 INFO:teuthology.orchestra.run.smithi059.stdout:Install 12 Packages 2024-04-01T13:28:42.856 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:42.856 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 37 M 2024-04-01T13:28:42.856 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 209 M 2024-04-01T13:28:42.856 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-04-01T13:28:43.317 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-immutable-object-cache-19.0.0-2568.gafa73a 247 kB/s | 157 kB 00:00 2024-04-01T13:28:43.317 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:28:43.318 INFO:teuthology.orchestra.run.smithi087.stdout:Total 247 kB/s | 157 kB 00:00 2024-04-01T13:28:43.318 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-01T13:28:43.327 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-01T13:28:43.327 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-01T13:28:43.349 INFO:teuthology.orchestra.run.smithi059.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 208 kB/s | 32 kB 00:00 2024-04-01T13:28:43.379 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-01T13:28:43.380 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-01T13:28:43.382 INFO:teuthology.orchestra.run.smithi059.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 535 kB/s | 17 kB 00:00 2024-04-01T13:28:43.424 INFO:teuthology.orchestra.run.smithi059.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 842 kB/s | 34 kB 00:00 2024-04-01T13:28:43.474 INFO:teuthology.orchestra.run.smithi059.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-04-01T13:28:43.524 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-01T13:28:43.584 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2568.gafa73a9e. 1/1 2024-04-01T13:28:43.608 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2568.gafa73a9e. 1/1 2024-04-01T13:28:43.608 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T13:28:43.608 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-01T13:28:43.609 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:43.650 INFO:teuthology.orchestra.run.smithi059.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-04-01T13:28:43.708 INFO:teuthology.orchestra.run.smithi059.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-04-01T13:28:43.733 INFO:teuthology.orchestra.run.smithi059.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-04-01T13:28:44.025 INFO:teuthology.orchestra.run.smithi059.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 50 MB/s | 16 MB 00:00 2024-04-01T13:28:44.254 INFO:teuthology.orchestra.run.smithi059.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 869 kB/s | 195 kB 00:00 2024-04-01T13:28:44.376 INFO:teuthology.orchestra.run.smithi059.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-04-01T13:28:44.401 INFO:teuthology.orchestra.run.smithi059.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-04-01T13:28:44.575 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2568.gafa73a9e. 1/1 2024-04-01T13:28:44.575 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:44.575 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-04-01T13:28:44.575 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:28:44.575 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:44.575 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:28:44.831 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr 2024-04-01T13:28:45.405 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:44 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:28:45.502 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-mgr-2:19.0.0-2568.gafa73a9e.el9.x86_64 is already installed. 2024-04-01T13:28:45.551 INFO:teuthology.orchestra.run.smithi059.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-2 3.1 MB/s | 7.4 MB 00:02 2024-04-01T13:28:45.554 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:28:45.554 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:28:45.555 INFO:teuthology.orchestra.run.smithi059.stdout:Total 14 MB/s | 37 MB 00:02 2024-04-01T13:28:45.555 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-04-01T13:28:45.555 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:28:45.599 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-04-01T13:28:45.621 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-dashboard 2024-04-01T13:28:45.626 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-04-01T13:28:45.626 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-04-01T13:28:46.010 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-04-01T13:28:46.010 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-04-01T13:28:46.178 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:45 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:28:46.330 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:28:46.331 INFO:teuthology.orchestra.run.smithi087.stdout:====================================================================================== 2024-04-01T13:28:46.331 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-04-01T13:28:46.331 INFO:teuthology.orchestra.run.smithi087.stdout:====================================================================================== 2024-04-01T13:28:46.331 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-04-01T13:28:46.331 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2568.gafa73a9e.el9 ceph-noarch 3.5 M 2024-04-01T13:28:46.331 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-04-01T13:28:46.331 INFO:teuthology.orchestra.run.smithi087.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-01T13:28:46.331 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2568.gafa73a9e.el9 ceph-noarch 34 k 2024-04-01T13:28:46.332 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2568.gafa73a9e.el9 ceph-noarch 16 k 2024-04-01T13:28:46.332 INFO:teuthology.orchestra.run.smithi087.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-01T13:28:46.332 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-01T13:28:46.332 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-01T13:28:46.332 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-01T13:28:46.332 INFO:teuthology.orchestra.run.smithi087.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-01T13:28:46.332 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-01T13:28:46.332 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-01T13:28:46.332 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:46.332 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-01T13:28:46.332 INFO:teuthology.orchestra.run.smithi087.stdout:====================================================================================== 2024-04-01T13:28:46.332 INFO:teuthology.orchestra.run.smithi087.stdout:Install 11 Packages 2024-04-01T13:28:46.332 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:46.333 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 7.6 M 2024-04-01T13:28:46.333 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 100 M 2024-04-01T13:28:46.333 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-01T13:28:46.866 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-04-01T13:28:46.933 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-01T13:28:46.964 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-01T13:28:47.061 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-01T13:28:47.062 INFO:teuthology.orchestra.run.smithi087.stdout:(1/11): ceph-prometheus-alerts-19.0.0-2568.gafa 56 kB/s | 16 kB 00:00 2024-04-01T13:28:47.087 INFO:teuthology.orchestra.run.smithi087.stdout:(2/11): ceph-grafana-dashboards-19.0.0-2568.gaf 106 kB/s | 34 kB 00:00 2024-04-01T13:28:47.101 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-01T13:28:47.221 INFO:teuthology.orchestra.run.smithi087.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-04-01T13:28:47.285 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-01T13:28:47.316 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-01T13:28:47.395 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-01T13:28:47.454 INFO:teuthology.orchestra.run.smithi087.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.3 MB/s | 548 kB 00:00 2024-04-01T13:28:47.505 INFO:teuthology.orchestra.run.smithi087.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 387 kB/s | 19 kB 00:00 2024-04-01T13:28:47.596 INFO:teuthology.orchestra.run.smithi087.stdout:(6/11): protobuf-compiler-3.14.0-13.el9.x86_64. 1.7 MB/s | 863 kB 00:00 2024-04-01T13:28:47.638 INFO:teuthology.orchestra.run.smithi087.stdout:(7/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 15 MB/s | 2.0 MB 00:00 2024-04-01T13:28:47.664 INFO:teuthology.orchestra.run.smithi087.stdout:(8/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-04-01T13:28:47.681 INFO:teuthology.orchestra.run.smithi087.stdout:(9/11): python3-repoze-lru-0.7-16.el9.noarch.rp 736 kB/s | 31 kB 00:00 2024-04-01T13:28:47.714 INFO:teuthology.orchestra.run.smithi087.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-04-01T13:28:48.104 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-01T13:28:48.223 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-01T13:28:49.512 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-01T13:28:49.714 INFO:teuthology.orchestra.run.smithi087.stdout:(11/11): ceph-mgr-dashboard-19.0.0-2568.gafa73a 1.2 MB/s | 3.5 MB 00:02 2024-04-01T13:28:49.720 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:28:49.720 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.2 MB/s | 7.6 MB 00:03 2024-04-01T13:28:49.741 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-01T13:28:49.759 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-01T13:28:49.759 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-01T13:28:49.931 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-01T13:28:49.931 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-01T13:28:50.419 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-01T13:28:50.512 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-01T13:28:50.551 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-01T13:28:50.590 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-01T13:28:50.614 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2568.gafa73 12/12 2024-04-01T13:28:50.651 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-01T13:28:50.776 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-01T13:28:50.829 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-01T13:28:50.912 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-01T13:28:50.955 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-01T13:28:50.984 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-01T13:28:51.047 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2568.gafa73a9e.el9 9/11 2024-04-01T13:28:52.106 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2568.gafa73a9e.el 10/11 2024-04-01T13:28:52.150 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2568.gafa73a9e.el9.noa 11/11 2024-04-01T13:28:53.131 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2568.gafa73 12/12 2024-04-01T13:28:53.131 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2568.gafa73 1/12 2024-04-01T13:28:53.131 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-01T13:28:53.131 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-01T13:28:53.131 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-01T13:28:53.132 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-01T13:28:53.132 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-01T13:28:53.132 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-01T13:28:53.132 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-01T13:28:53.132 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-01T13:28:53.132 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-01T13:28:53.132 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-01T13:28:53.414 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2568.gafa73a9e.el9.noa 11/11 2024-04-01T13:28:53.414 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2568.gafa73a9e.el 1/11 2024-04-01T13:28:53.414 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2568.gafa73a9e.el9.noa 2/11 2024-04-01T13:28:53.414 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2568.gafa73a9e.el9 3/11 2024-04-01T13:28:53.415 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-01T13:28:53.415 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-01T13:28:53.415 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-01T13:28:53.415 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-01T13:28:53.415 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-01T13:28:53.415 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-01T13:28:53.415 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-01T13:28:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-01T13:28:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:53.610 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-04-01T13:28:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2568.gafa73a9e.el9.noarch 2024-04-01T13:28:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-01T13:28:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-01T13:28:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-01T13:28:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-01T13:28:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-01T13:28:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-01T13:28:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-01T13:28:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-01T13:28:53.611 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-01T13:28:53.611 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-01T13:28:53.611 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-01T13:28:53.611 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:53.611 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:28:53.753 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-01T13:28:53.753 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:53.753 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-04-01T13:28:53.753 INFO:teuthology.orchestra.run.smithi087.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-01T13:28:53.753 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards-2:19.0.0-2568.gafa73a9e.el9.noarch 2024-04-01T13:28:53.753 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard-2:19.0.0-2568.gafa73a9e.el9.noarch 2024-04-01T13:28:53.754 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts-2:19.0.0-2568.gafa73a9e.el9.noarch 2024-04-01T13:28:53.754 INFO:teuthology.orchestra.run.smithi087.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-01T13:28:53.754 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-01T13:28:53.754 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-01T13:28:53.754 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-01T13:28:53.754 INFO:teuthology.orchestra.run.smithi087.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-01T13:28:53.754 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-01T13:28:53.754 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-01T13:28:53.754 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:53.754 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:28:53.997 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-01T13:28:54.047 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-rook 2024-04-01T13:28:54.559 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:53 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:28:54.624 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:59 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:28:54.712 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:28:54.713 INFO:teuthology.orchestra.run.smithi087.stdout:============================================================================================ 2024-04-01T13:28:54.713 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-04-01T13:28:54.713 INFO:teuthology.orchestra.run.smithi087.stdout:============================================================================================ 2024-04-01T13:28:54.713 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-04-01T13:28:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2568.gafa73a9e.el9 ceph-noarch 7.4 M 2024-04-01T13:28:54.714 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-04-01T13:28:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-01T13:28:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-01T13:28:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-01T13:28:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-01T13:28:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-01T13:28:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-01T13:28:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-01T13:28:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-01T13:28:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-01T13:28:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-01T13:28:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-01T13:28:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:54.714 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-01T13:28:54.715 INFO:teuthology.orchestra.run.smithi087.stdout:============================================================================================ 2024-04-01T13:28:54.715 INFO:teuthology.orchestra.run.smithi087.stdout:Install 12 Packages 2024-04-01T13:28:54.715 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:28:54.715 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 37 M 2024-04-01T13:28:54.715 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 209 M 2024-04-01T13:28:54.715 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-01T13:28:54.782 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:28:54.783 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2024-04-01T13:28:54.783 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-04-01T13:28:54.783 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2024-04-01T13:28:54.783 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-04-01T13:28:54.783 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook noarch 2:19.0.0-2568.gafa73a9e.el9 ceph-noarch 49 k 2024-04-01T13:28:54.783 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-04-01T13:28:54.784 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-01T13:28:54.784 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-01T13:28:54.784 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-01T13:28:54.784 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-01T13:28:54.784 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-01T13:28:54.784 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-01T13:28:54.784 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-01T13:28:54.784 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-01T13:28:54.784 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-01T13:28:54.784 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-01T13:28:54.784 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-01T13:28:54.784 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-01T13:28:54.784 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:54.784 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-04-01T13:28:54.785 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2024-04-01T13:28:54.785 INFO:teuthology.orchestra.run.smithi059.stdout:Install 13 Packages 2024-04-01T13:28:54.785 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:28:54.785 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 2.0 M 2024-04-01T13:28:54.785 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 26 M 2024-04-01T13:28:54.785 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-04-01T13:28:54.945 INFO:teuthology.orchestra.run.smithi087.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 300 kB/s | 32 kB 00:00 2024-04-01T13:28:54.980 INFO:teuthology.orchestra.run.smithi059.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 170 kB/s | 24 kB 00:00 2024-04-01T13:28:55.004 INFO:teuthology.orchestra.run.smithi087.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 490 kB/s | 17 kB 00:00 2024-04-01T13:28:55.005 INFO:teuthology.orchestra.run.smithi059.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 107 kB/s | 18 kB 00:00 2024-04-01T13:28:55.046 INFO:teuthology.orchestra.run.smithi087.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 826 kB/s | 34 kB 00:00 2024-04-01T13:28:55.080 INFO:teuthology.orchestra.run.smithi059.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-04-01T13:28:55.106 INFO:teuthology.orchestra.run.smithi059.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 132 kB 00:00 2024-04-01T13:28:55.122 INFO:teuthology.orchestra.run.smithi059.stdout:(5/13): ceph-mgr-rook-19.0.0-2568.gafa73a9e.el9 172 kB/s | 49 kB 00:00 2024-04-01T13:28:55.173 INFO:teuthology.orchestra.run.smithi059.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.2 MB/s | 211 kB 00:00 2024-04-01T13:28:55.190 INFO:teuthology.orchestra.run.smithi059.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 510 kB/s | 43 kB 00:00 2024-04-01T13:28:55.380 INFO:teuthology.orchestra.run.smithi087.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-04-01T13:28:55.580 INFO:teuthology.orchestra.run.smithi087.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2024-04-01T13:28:55.622 INFO:teuthology.orchestra.run.smithi087.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-04-01T13:28:55.839 INFO:teuthology.orchestra.run.smithi087.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.0 MB/s | 3.0 MB 00:00 2024-04-01T13:28:56.123 INFO:teuthology.orchestra.run.smithi087.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-04-01T13:28:56.248 INFO:teuthology.orchestra.run.smithi059.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 14 kB/s | 15 kB 00:01 2024-04-01T13:28:56.265 INFO:teuthology.orchestra.run.smithi059.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 28 kB/s | 32 kB 00:01 2024-04-01T13:28:56.306 INFO:teuthology.orchestra.run.smithi087.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-04-01T13:28:56.331 INFO:teuthology.orchestra.run.smithi087.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 937 kB/s | 195 kB 00:00 2024-04-01T13:28:56.381 INFO:teuthology.orchestra.run.smithi087.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-04-01T13:28:56.941 INFO:teuthology.orchestra.run.smithi059.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 116 kB/s | 203 kB 00:01 2024-04-01T13:28:57.099 INFO:teuthology.orchestra.run.smithi087.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-2 3.3 MB/s | 7.4 MB 00:02 2024-04-01T13:28:57.101 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:28:57.101 INFO:teuthology.orchestra.run.smithi087.stdout:Total 16 MB/s | 37 MB 00:02 2024-04-01T13:28:57.146 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-01T13:28:57.171 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-01T13:28:57.171 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-01T13:28:57.391 INFO:teuthology.orchestra.run.smithi059.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 52 kB/s | 59 kB 00:01 2024-04-01T13:28:57.525 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-01T13:28:57.526 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-01T13:28:58.091 INFO:teuthology.orchestra.run.smithi059.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 78 kB/s | 90 kB 00:01 2024-04-01T13:28:58.141 INFO:teuthology.orchestra.run.smithi059.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 552 kB/s | 1.0 MB 00:01 2024-04-01T13:28:58.143 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:28:58.144 INFO:teuthology.orchestra.run.smithi059.stdout:Total 624 kB/s | 2.0 MB 00:03 2024-04-01T13:28:58.144 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-04-01T13:28:58.160 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-04-01T13:28:58.160 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-04-01T13:28:58.236 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-01T13:28:58.264 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-04-01T13:28:58.264 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-04-01T13:28:58.313 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-01T13:28:58.344 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-01T13:28:58.438 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-01T13:28:58.481 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-01T13:28:58.658 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-01T13:28:58.719 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-01T13:28:58.799 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-01T13:28:58.804 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-04-01T13:28:58.973 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-01T13:28:59.027 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-01T13:28:59.083 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-01T13:28:59.127 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-01T13:28:59.177 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-01T13:28:59.274 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-01T13:28:59.385 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-01T13:28:59.437 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-01T13:28:59.462 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-01T13:28:59.566 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-01T13:29:00.196 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-01T13:29:00.250 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-01T13:29:00.284 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-01T13:29:00.332 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-01T13:29:00.361 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-rook-2:19.0.0-2568.gafa73a9e.el9.noarch 13/13 2024-04-01T13:29:00.811 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-01T13:29:01.190 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2568.gafa73a9e.el9.noarch 13/13 2024-04-01T13:29:01.190 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2568.gafa73a9e.el9.noarch 1/13 2024-04-01T13:29:01.190 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-01T13:29:01.190 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-01T13:29:01.191 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-01T13:29:01.191 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-01T13:29:01.191 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-01T13:29:01.191 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-01T13:29:01.191 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-01T13:29:01.191 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-01T13:29:01.191 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-01T13:29:01.191 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-01T13:29:01.191 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-01T13:29:01.557 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-01T13:29:01.557 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:01.557 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-04-01T13:29:01.557 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook-2:19.0.0-2568.gafa73a9e.el9.noarch 2024-04-01T13:29:01.557 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-01T13:29:01.558 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-01T13:29:01.558 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-01T13:29:01.558 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-01T13:29:01.558 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-01T13:29:01.558 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-01T13:29:01.558 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-01T13:29:01.558 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-01T13:29:01.558 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-01T13:29:01.558 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-01T13:29:01.558 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-01T13:29:01.558 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-01T13:29:01.558 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:01.558 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:29:01.773 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-cephadm 2024-04-01T13:29:01.827 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-01T13:29:01.862 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2568.gafa73 12/12 2024-04-01T13:29:02.348 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:07 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:29:02.506 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:29:02.507 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2024-04-01T13:29:02.507 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-04-01T13:29:02.507 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2024-04-01T13:29:02.507 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-04-01T13:29:02.507 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2568.gafa73a9e.el9 ceph-noarch 141 k 2024-04-01T13:29:02.507 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-04-01T13:29:02.507 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-01T13:29:02.507 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-01T13:29:02.508 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-01T13:29:02.508 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-01T13:29:02.508 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-01T13:29:02.508 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:02.508 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-04-01T13:29:02.508 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2024-04-01T13:29:02.508 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2024-04-01T13:29:02.508 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:02.509 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 6.8 M 2024-04-01T13:29:02.509 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 33 M 2024-04-01T13:29:02.509 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-04-01T13:29:03.001 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-04-01T13:29:03.093 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2568.gafa73a9e.e 544 kB/s | 141 kB 00:00 2024-04-01T13:29:03.401 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 187 kB/s | 58 kB 00:00 2024-04-01T13:29:03.577 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 953 kB/s | 548 kB 00:00 2024-04-01T13:29:03.593 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 266 kB/s | 51 kB 00:00 2024-04-01T13:29:04.127 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.5 MB/s | 5.8 MB 00:01 2024-04-01T13:29:04.131 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:29:04.131 INFO:teuthology.orchestra.run.smithi059.stdout:Total 4.2 MB/s | 6.8 MB 00:01 2024-04-01T13:29:04.147 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-04-01T13:29:04.159 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-04-01T13:29:04.159 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-04-01T13:29:04.255 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-04-01T13:29:04.256 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-04-01T13:29:04.352 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2568.gafa73 12/12 2024-04-01T13:29:04.352 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2568.gafa73 1/12 2024-04-01T13:29:04.353 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-01T13:29:04.353 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-01T13:29:04.353 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-01T13:29:04.353 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-01T13:29:04.353 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-01T13:29:04.353 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-01T13:29:04.353 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-01T13:29:04.353 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-01T13:29:04.354 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-01T13:29:04.354 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-01T13:29:04.553 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-04-01T13:29:04.724 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-01T13:29:04.788 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-01T13:29:04.790 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-01T13:29:04.790 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:04.790 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-04-01T13:29:04.790 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2568.gafa73a9e.el9.noarch 2024-04-01T13:29:04.790 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-01T13:29:04.790 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-01T13:29:04.790 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-01T13:29:04.790 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-01T13:29:04.791 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-01T13:29:04.791 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-01T13:29:04.791 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-01T13:29:04.791 INFO:teuthology.orchestra.run.smithi087.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-01T13:29:04.791 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-01T13:29:04.791 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-01T13:29:04.791 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-01T13:29:04.791 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:04.791 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:29:05.210 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-rook 2024-04-01T13:29:05.342 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-01T13:29:05.425 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-01T13:29:05.495 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-01T13:29:05.524 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2568.gafa73a9e.el9.noarch 6/6 2024-04-01T13:29:05.784 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:04 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:29:05.935 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:29:05.936 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2024-04-01T13:29:05.936 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-04-01T13:29:05.936 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2024-04-01T13:29:05.936 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-04-01T13:29:05.936 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook noarch 2:19.0.0-2568.gafa73a9e.el9 ceph-noarch 49 k 2024-04-01T13:29:05.936 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-04-01T13:29:05.936 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-01T13:29:05.937 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-01T13:29:05.937 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-01T13:29:05.937 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-01T13:29:05.937 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-01T13:29:05.937 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-01T13:29:05.937 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-01T13:29:05.937 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-01T13:29:05.937 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-01T13:29:05.937 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-01T13:29:05.937 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-01T13:29:05.937 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-01T13:29:05.937 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:05.937 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-01T13:29:05.937 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2024-04-01T13:29:05.938 INFO:teuthology.orchestra.run.smithi087.stdout:Install 13 Packages 2024-04-01T13:29:05.938 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:05.938 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 2.0 M 2024-04-01T13:29:05.938 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 26 M 2024-04-01T13:29:05.938 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-01T13:29:06.245 INFO:teuthology.orchestra.run.smithi087.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 158 kB/s | 18 kB 00:00 2024-04-01T13:29:06.262 INFO:teuthology.orchestra.run.smithi087.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 185 kB/s | 24 kB 00:00 2024-04-01T13:29:06.312 INFO:teuthology.orchestra.run.smithi087.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-04-01T13:29:06.320 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2568.gafa73a9e.el9.noarch 6/6 2024-04-01T13:29:06.320 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2568.gafa73a9e.el9.noarch 1/6 2024-04-01T13:29:06.320 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-01T13:29:06.320 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-01T13:29:06.320 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-01T13:29:06.321 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-01T13:29:06.337 INFO:teuthology.orchestra.run.smithi087.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-01T13:29:06.363 INFO:teuthology.orchestra.run.smithi087.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-04-01T13:29:06.388 INFO:teuthology.orchestra.run.smithi087.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-04-01T13:29:06.538 INFO:teuthology.orchestra.run.smithi087.stdout:(7/13): ceph-mgr-rook-19.0.0-2568.gafa73a9e.el9 121 kB/s | 49 kB 00:00 2024-04-01T13:29:06.621 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-01T13:29:06.621 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:06.621 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-04-01T13:29:06.621 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm-2:19.0.0-2568.gafa73a9e.el9.noarch 2024-04-01T13:29:06.621 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-01T13:29:06.621 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-01T13:29:06.621 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-01T13:29:06.621 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-01T13:29:06.621 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-01T13:29:06.621 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:06.622 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:29:06.664 INFO:teuthology.orchestra.run.smithi087.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2024-04-01T13:29:06.689 INFO:teuthology.orchestra.run.smithi087.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 50 kB/s | 15 kB 00:00 2024-04-01T13:29:06.839 INFO:teuthology.orchestra.run.smithi087.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 396 kB/s | 59 kB 00:00 2024-04-01T13:29:06.861 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-fuse 2024-04-01T13:29:06.964 INFO:teuthology.orchestra.run.smithi087.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 477 kB/s | 203 kB 00:00 2024-04-01T13:29:06.989 INFO:teuthology.orchestra.run.smithi087.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 597 kB/s | 90 kB 00:00 2024-04-01T13:29:07.023 INFO:teuthology.orchestra.run.smithi087.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.8 MB/s | 1.0 MB 00:00 2024-04-01T13:29:07.027 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:29:07.027 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-04-01T13:29:07.027 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-01T13:29:07.040 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-01T13:29:07.040 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-01T13:29:07.130 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-01T13:29:07.131 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-01T13:29:07.438 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:12 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:29:07.598 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:29:07.598 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:07.598 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-04-01T13:29:07.598 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:07.599 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-04-01T13:29:07.599 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 871 k 2024-04-01T13:29:07.599 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-04-01T13:29:07.599 INFO:teuthology.orchestra.run.smithi059.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-01T13:29:07.599 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:07.599 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-04-01T13:29:07.599 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:07.599 INFO:teuthology.orchestra.run.smithi059.stdout:Install 2 Packages 2024-04-01T13:29:07.599 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:07.600 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 949 k 2024-04-01T13:29:07.600 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 4.0 M 2024-04-01T13:29:07.600 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-04-01T13:29:07.631 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-01T13:29:07.797 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-01T13:29:07.828 INFO:teuthology.orchestra.run.smithi059.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 344 kB/s | 79 kB 00:00 2024-04-01T13:29:07.848 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-01T13:29:07.905 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-01T13:29:07.951 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-01T13:29:07.992 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-01T13:29:08.087 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-01T13:29:08.162 INFO:teuthology.orchestra.run.smithi059.stdout:(2/2): ceph-fuse-19.0.0-2568.gafa73a9e.el9.x86_ 1.5 MB/s | 871 kB 00:00 2024-04-01T13:29:08.163 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:29:08.163 INFO:teuthology.orchestra.run.smithi059.stdout:Total 1.6 MB/s | 949 kB 00:00 2024-04-01T13:29:08.163 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-04-01T13:29:08.188 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-04-01T13:29:08.188 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-04-01T13:29:08.195 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-01T13:29:08.243 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-01T13:29:08.263 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-04-01T13:29:08.263 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-04-01T13:29:08.447 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-04-01T13:29:08.593 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-01T13:29:08.673 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-fuse-2:19.0.0-2568.gafa73a9e.el9.x86_64 2/2 2024-04-01T13:29:08.966 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-01T13:29:09.017 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-01T13:29:09.050 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-01T13:29:09.107 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-01T13:29:09.137 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-rook-2:19.0.0-2568.gafa73a9e.el9.noarch 13/13 2024-04-01T13:29:09.348 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2568.gafa73a9e.el9.x86_64 2/2 2024-04-01T13:29:09.348 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-fuse-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/2 2024-04-01T13:29:09.663 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-01T13:29:09.663 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:09.663 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-04-01T13:29:09.663 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse-2:19.0.0-2568.gafa73a9e.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-01T13:29:09.663 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:09.664 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:29:09.964 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2568.gafa73a9e.el9.noarch 13/13 2024-04-01T13:29:09.964 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2568.gafa73a9e.el9.noarch 1/13 2024-04-01T13:29:09.964 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-01T13:29:09.964 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-01T13:29:09.964 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-01T13:29:09.964 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-01T13:29:09.964 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-01T13:29:09.964 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-01T13:29:09.964 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-01T13:29:09.964 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-01T13:29:09.964 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-01T13:29:09.965 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-01T13:29:09.965 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-01T13:29:09.969 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-volume 2024-04-01T13:29:10.379 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-01T13:29:10.379 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:10.379 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-04-01T13:29:10.379 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook-2:19.0.0-2568.gafa73a9e.el9.noarch 2024-04-01T13:29:10.379 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-01T13:29:10.379 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-01T13:29:10.379 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-01T13:29:10.379 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-01T13:29:10.380 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-01T13:29:10.380 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-01T13:29:10.380 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-01T13:29:10.380 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-01T13:29:10.380 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-01T13:29:10.380 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-01T13:29:10.380 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-01T13:29:10.380 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-01T13:29:10.380 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:10.380 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:29:10.550 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:15 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:29:10.621 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-cephadm 2024-04-01T13:29:10.708 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:29:10.709 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:10.709 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-04-01T13:29:10.709 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:10.709 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-04-01T13:29:10.709 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume noarch 2:19.0.0-2568.gafa73a9e.el9 ceph-noarch 264 k 2024-04-01T13:29:10.709 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-04-01T13:29:10.709 INFO:teuthology.orchestra.run.smithi059.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-01T13:29:10.709 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:10.709 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-04-01T13:29:10.710 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:10.710 INFO:teuthology.orchestra.run.smithi059.stdout:Install 2 Packages 2024-04-01T13:29:10.710 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:10.710 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 332 k 2024-04-01T13:29:10.710 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 1.5 M 2024-04-01T13:29:10.710 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-04-01T13:29:10.899 INFO:teuthology.orchestra.run.smithi059.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 366 kB/s | 69 kB 00:00 2024-04-01T13:29:11.115 INFO:teuthology.orchestra.run.smithi059.stdout:(2/2): ceph-volume-19.0.0-2568.gafa73a9e.el9.no 651 kB/s | 264 kB 00:00 2024-04-01T13:29:11.116 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:29:11.117 INFO:teuthology.orchestra.run.smithi059.stdout:Total 817 kB/s | 332 kB 00:00 2024-04-01T13:29:11.117 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-04-01T13:29:11.126 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-04-01T13:29:11.127 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-04-01T13:29:11.169 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-04-01T13:29:11.169 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-04-01T13:29:11.181 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:10 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:29:11.333 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:29:11.333 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2024-04-01T13:29:11.333 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-04-01T13:29:11.334 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2024-04-01T13:29:11.334 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-04-01T13:29:11.334 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2568.gafa73a9e.el9 ceph-noarch 141 k 2024-04-01T13:29:11.334 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-04-01T13:29:11.334 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-01T13:29:11.334 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-01T13:29:11.334 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-01T13:29:11.334 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-01T13:29:11.334 INFO:teuthology.orchestra.run.smithi087.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-01T13:29:11.334 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:11.334 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-01T13:29:11.335 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2024-04-01T13:29:11.335 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2024-04-01T13:29:11.335 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:11.335 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 6.8 M 2024-04-01T13:29:11.335 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 33 M 2024-04-01T13:29:11.335 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-01T13:29:11.388 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-04-01T13:29:11.573 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-01T13:29:11.603 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-volume-2:19.0.0-2568.gafa73a9e.el9.noarch 2/2 2024-04-01T13:29:11.619 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-volume-2:19.0.0-2568.gafa73a9e.el9.noarch 2/2 2024-04-01T13:29:11.619 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T13:29:11.619 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-01T13:29:11.619 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:11.634 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-04-01T13:29:11.743 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2568.gafa73a9e.e 478 kB/s | 141 kB 00:00 2024-04-01T13:29:11.868 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-04-01T13:29:12.393 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-volume-2:19.0.0-2568.gafa73a9e.el9.noarch 1/2 2024-04-01T13:29:12.502 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 80 kB/s | 51 kB 00:00 2024-04-01T13:29:12.527 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 74 kB/s | 58 kB 00:00 2024-04-01T13:29:12.753 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-01T13:29:12.753 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:12.753 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-04-01T13:29:12.753 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume-2:19.0.0-2568.gafa73a9e.el9.noarch 2024-04-01T13:29:12.753 INFO:teuthology.orchestra.run.smithi059.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-01T13:29:12.753 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:12.754 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:29:13.044 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados-devel 2024-04-01T13:29:13.622 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:18 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:29:13.779 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:29:13.780 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:13.780 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-04-01T13:29:13.780 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:13.780 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-04-01T13:29:13.780 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 125 k 2024-04-01T13:29:13.780 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:13.781 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-04-01T13:29:13.781 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:13.781 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-04-01T13:29:13.781 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:13.781 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 125 k 2024-04-01T13:29:13.781 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 445 k 2024-04-01T13:29:13.781 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-04-01T13:29:13.990 INFO:teuthology.orchestra.run.smithi059.stdout:librados-devel-19.0.0-2568.gafa73a9e.el9.x86_64 599 kB/s | 125 kB 00:00 2024-04-01T13:29:13.990 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:29:13.991 INFO:teuthology.orchestra.run.smithi059.stdout:Total 595 kB/s | 125 kB 00:00 2024-04-01T13:29:13.991 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-04-01T13:29:13.997 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-04-01T13:29:13.997 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-04-01T13:29:14.061 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-04-01T13:29:14.062 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-04-01T13:29:14.215 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-04-01T13:29:14.316 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librados-devel-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:14.419 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 197 kB/s | 548 kB 00:02 2024-04-01T13:29:14.420 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:29:14.420 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.2 MB/s | 6.8 MB 00:03 2024-04-01T13:29:14.436 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-01T13:29:14.448 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-01T13:29:14.448 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-01T13:29:14.540 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-01T13:29:14.541 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-01T13:29:14.806 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados-devel-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:14.837 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-01T13:29:15.019 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-01T13:29:15.070 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-01T13:29:15.098 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados-devel-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:15.098 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:15.098 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-04-01T13:29:15.098 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:29:15.098 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:15.099 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:29:15.313 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs2 2024-04-01T13:29:15.592 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-01T13:29:15.681 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-01T13:29:15.753 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-01T13:29:15.782 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2568.gafa73a9e.el9.noarch 6/6 2024-04-01T13:29:15.900 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:20 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:29:16.004 INFO:teuthology.orchestra.run.smithi059.stdout:Package libcephfs2-2:19.0.0-2568.gafa73a9e.el9.x86_64 is already installed. 2024-04-01T13:29:16.059 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:29:16.060 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-04-01T13:29:16.060 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:29:16.122 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs-devel 2024-04-01T13:29:16.579 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2568.gafa73a9e.el9.noarch 6/6 2024-04-01T13:29:16.579 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2568.gafa73a9e.el9.noarch 1/6 2024-04-01T13:29:16.579 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-01T13:29:16.579 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-01T13:29:16.579 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-01T13:29:16.579 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-01T13:29:16.696 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:21 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:29:16.855 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:29:16.856 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:16.856 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-04-01T13:29:16.856 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:16.856 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-04-01T13:29:16.856 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 32 k 2024-04-01T13:29:16.856 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:16.857 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-04-01T13:29:16.857 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:16.857 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-04-01T13:29:16.857 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:16.857 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 32 k 2024-04-01T13:29:16.857 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 142 k 2024-04-01T13:29:16.857 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-04-01T13:29:16.859 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-01T13:29:16.859 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:16.859 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-04-01T13:29:16.859 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm-2:19.0.0-2568.gafa73a9e.el9.noarch 2024-04-01T13:29:16.859 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-01T13:29:16.859 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-01T13:29:16.859 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-01T13:29:16.859 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-01T13:29:16.859 INFO:teuthology.orchestra.run.smithi087.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-01T13:29:16.860 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:16.860 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:29:17.101 INFO:teuthology.orchestra.run.smithi059.stdout:libcephfs-devel-19.0.0-2568.gafa73a9e.el9.x86_6 130 kB/s | 32 kB 00:00 2024-04-01T13:29:17.102 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:29:17.102 INFO:teuthology.orchestra.run.smithi059.stdout:Total 129 kB/s | 32 kB 00:00 2024-04-01T13:29:17.102 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-04-01T13:29:17.103 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-fuse 2024-04-01T13:29:17.105 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-04-01T13:29:17.106 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-04-01T13:29:17.137 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-04-01T13:29:17.137 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-04-01T13:29:17.232 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-04-01T13:29:17.361 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs-devel-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:17.662 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:16 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:29:17.731 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:17.814 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:29:17.815 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:17.815 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-04-01T13:29:17.815 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:17.815 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-04-01T13:29:17.815 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 871 k 2024-04-01T13:29:17.815 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-04-01T13:29:17.815 INFO:teuthology.orchestra.run.smithi087.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-01T13:29:17.815 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:17.815 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-01T13:29:17.815 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:17.816 INFO:teuthology.orchestra.run.smithi087.stdout:Install 2 Packages 2024-04-01T13:29:17.816 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:17.816 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 949 k 2024-04-01T13:29:17.816 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 4.0 M 2024-04-01T13:29:17.816 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-01T13:29:17.979 INFO:teuthology.orchestra.run.smithi087.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 484 kB/s | 79 kB 00:00 2024-04-01T13:29:17.985 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs-devel-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:17.985 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:17.986 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-04-01T13:29:17.986 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:29:17.986 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:17.986 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:29:18.151 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados2 2024-04-01T13:29:18.362 INFO:teuthology.orchestra.run.smithi087.stdout:(2/2): ceph-fuse-19.0.0-2568.gafa73a9e.el9.x86_ 1.6 MB/s | 871 kB 00:00 2024-04-01T13:29:18.363 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:29:18.363 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.7 MB/s | 949 kB 00:00 2024-04-01T13:29:18.363 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-01T13:29:18.386 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-01T13:29:18.386 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-01T13:29:18.459 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-01T13:29:18.459 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-01T13:29:18.642 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-01T13:29:18.726 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:23 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:29:18.797 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-01T13:29:18.828 INFO:teuthology.orchestra.run.smithi059.stdout:Package librados2-2:19.0.0-2568.gafa73a9e.el9.x86_64 is already installed. 2024-04-01T13:29:18.861 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-fuse-2:19.0.0-2568.gafa73a9e.el9.x86_64 2/2 2024-04-01T13:29:18.884 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:29:18.885 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-04-01T13:29:18.885 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:29:18.950 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librbd1 2024-04-01T13:29:19.526 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:24 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:29:19.577 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2568.gafa73a9e.el9.x86_64 2/2 2024-04-01T13:29:19.577 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-fuse-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/2 2024-04-01T13:29:19.629 INFO:teuthology.orchestra.run.smithi059.stdout:Package librbd1-2:19.0.0-2568.gafa73a9e.el9.x86_64 is already installed. 2024-04-01T13:29:19.685 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:29:19.686 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-04-01T13:29:19.686 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:29:19.750 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rados 2024-04-01T13:29:19.888 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-01T13:29:19.888 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:19.889 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-04-01T13:29:19.889 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse-2:19.0.0-2568.gafa73a9e.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-01T13:29:19.889 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:19.889 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:29:20.159 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-volume 2024-04-01T13:29:20.330 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:25 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:29:20.432 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rados-2:19.0.0-2568.gafa73a9e.el9.x86_64 is already installed. 2024-04-01T13:29:20.488 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:29:20.489 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-04-01T13:29:20.489 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:29:20.550 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rgw 2024-04-01T13:29:20.740 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:19 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:29:20.894 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:29:20.895 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:20.895 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-04-01T13:29:20.895 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:20.895 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-04-01T13:29:20.895 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume noarch 2:19.0.0-2568.gafa73a9e.el9 ceph-noarch 264 k 2024-04-01T13:29:20.895 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-04-01T13:29:20.896 INFO:teuthology.orchestra.run.smithi087.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-01T13:29:20.896 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:20.896 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-01T13:29:20.896 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:20.896 INFO:teuthology.orchestra.run.smithi087.stdout:Install 2 Packages 2024-04-01T13:29:20.896 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:20.896 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 332 k 2024-04-01T13:29:20.896 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 1.5 M 2024-04-01T13:29:20.896 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-01T13:29:21.041 INFO:teuthology.orchestra.run.smithi087.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 477 kB/s | 69 kB 00:00 2024-04-01T13:29:21.127 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:26 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:29:21.229 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rgw-2:19.0.0-2568.gafa73a9e.el9.x86_64 is already installed. 2024-04-01T13:29:21.284 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:29:21.285 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-04-01T13:29:21.285 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:29:21.299 INFO:teuthology.orchestra.run.smithi087.stdout:(2/2): ceph-volume-19.0.0-2568.gafa73a9e.el9.no 655 kB/s | 264 kB 00:00 2024-04-01T13:29:21.300 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:29:21.300 INFO:teuthology.orchestra.run.smithi087.stdout:Total 821 kB/s | 332 kB 00:00 2024-04-01T13:29:21.301 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-01T13:29:21.309 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-01T13:29:21.309 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-01T13:29:21.342 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-01T13:29:21.343 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-01T13:29:21.358 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-cephfs 2024-04-01T13:29:21.566 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-01T13:29:21.748 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-01T13:29:21.785 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-volume-2:19.0.0-2568.gafa73a9e.el9.noarch 2/2 2024-04-01T13:29:21.801 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-volume-2:19.0.0-2568.gafa73a9e.el9.noarch 2/2 2024-04-01T13:29:21.801 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T13:29:21.801 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-01T13:29:21.801 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:21.930 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:26 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:29:22.031 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-cephfs-2:19.0.0-2568.gafa73a9e.el9.x86_64 is already installed. 2024-04-01T13:29:22.085 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:29:22.087 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-04-01T13:29:22.087 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:29:22.149 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rbd 2024-04-01T13:29:22.538 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-volume-2:19.0.0-2568.gafa73a9e.el9.noarch 1/2 2024-04-01T13:29:22.729 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:27 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:29:22.831 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rbd-2:19.0.0-2568.gafa73a9e.el9.x86_64 is already installed. 2024-04-01T13:29:22.887 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:29:22.888 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-04-01T13:29:22.888 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:29:22.900 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-01T13:29:22.900 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:22.900 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-04-01T13:29:22.900 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume-2:19.0.0-2568.gafa73a9e.el9.noarch 2024-04-01T13:29:22.900 INFO:teuthology.orchestra.run.smithi087.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-01T13:29:22.900 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:22.901 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:29:22.949 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-fuse 2024-04-01T13:29:23.187 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados-devel 2024-04-01T13:29:23.526 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:28 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:29:23.685 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:29:23.686 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:23.686 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-04-01T13:29:23.686 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:23.686 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-04-01T13:29:23.686 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 90 k 2024-04-01T13:29:23.686 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:23.686 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-04-01T13:29:23.686 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:23.686 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-04-01T13:29:23.686 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:23.686 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 90 k 2024-04-01T13:29:23.686 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 313 k 2024-04-01T13:29:23.687 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-04-01T13:29:23.763 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:22 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:29:23.904 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-fuse-19.0.0-2568.gafa73a9e.el9.x86_64.rpm 415 kB/s | 90 kB 00:00 2024-04-01T13:29:23.904 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:29:23.904 INFO:teuthology.orchestra.run.smithi059.stdout:Total 412 kB/s | 90 kB 00:00 2024-04-01T13:29:23.905 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-04-01T13:29:23.912 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:29:23.913 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:23.913 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-04-01T13:29:23.913 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:23.913 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-04-01T13:29:23.913 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 125 k 2024-04-01T13:29:23.913 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:23.913 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-01T13:29:23.913 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:23.913 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-04-01T13:29:23.914 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:23.914 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 125 k 2024-04-01T13:29:23.914 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 445 k 2024-04-01T13:29:23.914 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-01T13:29:23.915 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-04-01T13:29:23.915 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-04-01T13:29:23.973 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-04-01T13:29:23.973 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-04-01T13:29:24.118 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-04-01T13:29:24.214 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-fuse-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:24.405 INFO:teuthology.orchestra.run.smithi087.stdout:librados-devel-19.0.0-2568.gafa73a9e.el9.x86_64 255 kB/s | 125 kB 00:00 2024-04-01T13:29:24.405 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:29:24.406 INFO:teuthology.orchestra.run.smithi087.stdout:Total 254 kB/s | 125 kB 00:00 2024-04-01T13:29:24.406 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-01T13:29:24.411 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-01T13:29:24.411 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-01T13:29:24.472 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-01T13:29:24.472 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-01T13:29:24.596 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:24.626 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-01T13:29:24.721 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librados-devel-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:24.869 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-fuse-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:24.869 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:24.869 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-04-01T13:29:24.869 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:29:24.869 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:24.869 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:29:25.079 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-mirror 2024-04-01T13:29:25.151 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados-devel-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:25.443 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados-devel-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:25.443 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:25.443 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-04-01T13:29:25.443 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:29:25.444 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:25.444 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:29:25.656 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:30 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:29:25.718 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs2 2024-04-01T13:29:25.813 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:29:25.814 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:25.814 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-04-01T13:29:25.814 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:25.814 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-04-01T13:29:25.814 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 3.4 M 2024-04-01T13:29:25.814 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:25.814 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-04-01T13:29:25.814 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:25.814 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-04-01T13:29:25.814 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:25.814 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 3.4 M 2024-04-01T13:29:25.815 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 29 M 2024-04-01T13:29:25.815 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-04-01T13:29:26.293 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:25 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:29:26.394 INFO:teuthology.orchestra.run.smithi087.stdout:Package libcephfs2-2:19.0.0-2568.gafa73a9e.el9.x86_64 is already installed. 2024-04-01T13:29:26.446 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:29:26.447 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-04-01T13:29:26.447 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:29:26.525 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs-devel 2024-04-01T13:29:26.923 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-mirror-19.0.0-2568.gafa73a9e.el9.x86_64.rpm 3.1 MB/s | 3.4 MB 00:01 2024-04-01T13:29:26.923 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:29:26.924 INFO:teuthology.orchestra.run.smithi059.stdout:Total 3.1 MB/s | 3.4 MB 00:01 2024-04-01T13:29:26.924 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-04-01T13:29:26.938 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-04-01T13:29:26.938 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-04-01T13:29:27.020 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-04-01T13:29:27.020 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-04-01T13:29:27.087 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:26 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:29:27.237 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:29:27.238 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:27.238 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-04-01T13:29:27.238 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:27.238 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-04-01T13:29:27.238 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 32 k 2024-04-01T13:29:27.238 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:27.238 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-01T13:29:27.238 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:27.238 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-04-01T13:29:27.238 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:27.238 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 32 k 2024-04-01T13:29:27.239 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 142 k 2024-04-01T13:29:27.239 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-01T13:29:27.415 INFO:teuthology.orchestra.run.smithi087.stdout:libcephfs-devel-19.0.0-2568.gafa73a9e.el9.x86_6 179 kB/s | 32 kB 00:00 2024-04-01T13:29:27.416 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:29:27.416 INFO:teuthology.orchestra.run.smithi087.stdout:Total 178 kB/s | 32 kB 00:00 2024-04-01T13:29:27.416 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-01T13:29:27.420 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-01T13:29:27.420 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-01T13:29:27.451 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-01T13:29:27.452 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-01T13:29:27.546 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-01T13:29:27.606 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-04-01T13:29:27.647 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs-devel-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:27.674 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-mirror-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:27.698 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:27.698 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T13:29:27.698 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-01T13:29:27.698 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-01T13:29:27.698 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-01T13:29:27.698 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:27.981 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:28.241 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs-devel-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:28.241 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:28.241 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-04-01T13:29:28.241 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:29:28.242 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:28.242 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:29:28.416 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados2 2024-04-01T13:29:28.969 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-mirror-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:28.970 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:28.970 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-04-01T13:29:28.970 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:29:28.970 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:28.970 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:29:28.979 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:27 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:29:29.079 INFO:teuthology.orchestra.run.smithi087.stdout:Package librados2-2:19.0.0-2568.gafa73a9e.el9.x86_64 is already installed. 2024-04-01T13:29:29.131 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:29:29.132 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-04-01T13:29:29.132 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:29:29.208 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librbd1 2024-04-01T13:29:29.255 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-nbd 2024-04-01T13:29:29.773 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:28 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:29:29.838 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:34 ago on Mon 01 Apr 2024 01:26:55 PM UTC. 2024-04-01T13:29:29.872 INFO:teuthology.orchestra.run.smithi087.stdout:Package librbd1-2:19.0.0-2568.gafa73a9e.el9.x86_64 is already installed. 2024-04-01T13:29:29.925 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:29:29.926 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-04-01T13:29:29.926 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:29:29.998 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-04-01T13:29:29.998 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:29.998 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-04-01T13:29:29.999 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:29.999 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-04-01T13:29:29.999 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 175 k 2024-04-01T13:29:29.999 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:29.999 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-04-01T13:29:29.999 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-04-01T13:29:29.999 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-04-01T13:29:29.999 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:29.999 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 175 k 2024-04-01T13:29:29.999 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 673 k 2024-04-01T13:29:29.999 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-04-01T13:29:30.000 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rados 2024-04-01T13:29:30.209 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-nbd-19.0.0-2568.gafa73a9e.el9.x86_64.rpm 837 kB/s | 175 kB 00:00 2024-04-01T13:29:30.210 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:29:30.210 INFO:teuthology.orchestra.run.smithi059.stdout:Total 830 kB/s | 175 kB 00:00 2024-04-01T13:29:30.210 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-04-01T13:29:30.221 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-04-01T13:29:30.221 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-04-01T13:29:30.279 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-04-01T13:29:30.280 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-04-01T13:29:30.432 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-04-01T13:29:30.553 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-nbd-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:30.564 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:29 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:29:30.663 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rados-2:19.0.0-2568.gafa73a9e.el9.x86_64 is already installed. 2024-04-01T13:29:30.715 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:29:30.716 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-04-01T13:29:30.716 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:29:30.788 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rgw 2024-04-01T13:29:31.026 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:31.325 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-nbd-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:31.326 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:31.326 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-04-01T13:29:31.326 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:29:31.326 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:31.326 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-04-01T13:29:31.356 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:30 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:29:31.454 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rgw-2:19.0.0-2568.gafa73a9e.el9.x86_64 is already installed. 2024-04-01T13:29:31.506 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:29:31.507 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-04-01T13:29:31.507 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:29:31.555 DEBUG:teuthology.parallel:result is None 2024-04-01T13:29:31.578 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-cephfs 2024-04-01T13:29:32.138 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:31 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:29:32.235 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-cephfs-2:19.0.0-2568.gafa73a9e.el9.x86_64 is already installed. 2024-04-01T13:29:32.286 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:29:32.287 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-04-01T13:29:32.287 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:29:32.356 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rbd 2024-04-01T13:29:32.924 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:31 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:29:33.022 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rbd-2:19.0.0-2568.gafa73a9e.el9.x86_64 is already installed. 2024-04-01T13:29:33.074 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:29:33.075 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-04-01T13:29:33.075 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:29:33.157 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-fuse 2024-04-01T13:29:33.759 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:32 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:29:33.920 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:29:33.920 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:33.920 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-04-01T13:29:33.920 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:33.920 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-04-01T13:29:33.921 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 90 k 2024-04-01T13:29:33.921 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:33.921 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-01T13:29:33.921 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:33.921 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-04-01T13:29:33.921 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:33.921 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 90 k 2024-04-01T13:29:33.921 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 313 k 2024-04-01T13:29:33.921 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-01T13:29:34.252 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-fuse-19.0.0-2568.gafa73a9e.el9.x86_64.rpm 273 kB/s | 90 kB 00:00 2024-04-01T13:29:34.253 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:29:34.253 INFO:teuthology.orchestra.run.smithi087.stdout:Total 271 kB/s | 90 kB 00:00 2024-04-01T13:29:34.253 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-01T13:29:34.262 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-01T13:29:34.262 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-01T13:29:34.320 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-01T13:29:34.320 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-01T13:29:34.474 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-01T13:29:34.585 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-fuse-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:34.994 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:35.280 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-fuse-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:35.281 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:35.281 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-04-01T13:29:35.281 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:29:35.281 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:35.281 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:29:35.501 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-mirror 2024-04-01T13:29:36.076 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:35 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:29:36.233 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:29:36.234 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:36.234 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-04-01T13:29:36.234 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:36.234 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-04-01T13:29:36.234 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 3.4 M 2024-04-01T13:29:36.234 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:36.235 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-01T13:29:36.235 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:36.235 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-04-01T13:29:36.235 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:36.235 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 3.4 M 2024-04-01T13:29:36.235 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 29 M 2024-04-01T13:29:36.235 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-01T13:29:36.938 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-mirror-19.0.0-2568.gafa73a9e.el9.x86_64.rpm 4.8 MB/s | 3.4 MB 00:00 2024-04-01T13:29:36.939 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:29:36.939 INFO:teuthology.orchestra.run.smithi087.stdout:Total 4.8 MB/s | 3.4 MB 00:00 2024-04-01T13:29:36.939 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-01T13:29:36.950 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-01T13:29:36.951 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-01T13:29:37.027 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-01T13:29:37.027 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-01T13:29:37.590 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-01T13:29:37.664 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-mirror-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:37.689 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:37.689 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T13:29:37.689 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-01T13:29:37.689 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-01T13:29:37.689 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-01T13:29:37.689 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:38.905 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-mirror-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:38.905 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:38.905 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-04-01T13:29:38.905 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:29:38.905 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:38.906 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:29:39.174 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-nbd 2024-04-01T13:29:39.758 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:38 ago on Mon 01 Apr 2024 01:27:01 PM UTC. 2024-04-01T13:29:39.910 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-01T13:29:39.911 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:39.911 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-04-01T13:29:39.911 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:39.911 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-04-01T13:29:39.911 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd x86_64 2:19.0.0-2568.gafa73a9e.el9 ceph 175 k 2024-04-01T13:29:39.912 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:39.912 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-01T13:29:39.912 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-01T13:29:39.912 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-04-01T13:29:39.912 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:39.912 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 175 k 2024-04-01T13:29:39.912 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 673 k 2024-04-01T13:29:39.912 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-01T13:29:40.242 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-nbd-19.0.0-2568.gafa73a9e.el9.x86_64.rpm 532 kB/s | 175 kB 00:00 2024-04-01T13:29:40.242 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-01T13:29:40.242 INFO:teuthology.orchestra.run.smithi087.stdout:Total 530 kB/s | 175 kB 00:00 2024-04-01T13:29:40.243 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-01T13:29:40.252 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-01T13:29:40.252 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-01T13:29:40.308 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-01T13:29:40.309 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-01T13:29:40.459 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-01T13:29:40.574 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-nbd-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:40.980 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:41.281 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-nbd-2:19.0.0-2568.gafa73a9e.el9.x86_64 1/1 2024-04-01T13:29:41.281 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:41.281 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-04-01T13:29:41.282 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd-2:19.0.0-2568.gafa73a9e.el9.x86_64 2024-04-01T13:29:41.282 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:41.282 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-01T13:29:41.527 DEBUG:teuthology.parallel:result is None 2024-04-01T13:29:41.528 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=afa73a9e21b8fb0610f8af7266f1975be37c0a69 2024-04-01T13:29:41.670 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-01T13:29:41.693 INFO:teuthology.orchestra.run.smithi059.stdout:19.0.0-2568.gafa73a9e.el9 2024-04-01T13:29:41.694 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2568.gafa73a9e.el9 2024-04-01T13:29:41.694 INFO:teuthology.task.install:The correct ceph version 19.0.0-2568.gafa73a9e is installed. 2024-04-01T13:29:41.696 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=afa73a9e21b8fb0610f8af7266f1975be37c0a69 2024-04-01T13:29:41.923 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-01T13:29:41.983 INFO:teuthology.orchestra.run.smithi087.stdout:19.0.0-2568.gafa73a9e.el9 2024-04-01T13:29:41.984 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2568.gafa73a9e.el9 2024-04-01T13:29:41.984 INFO:teuthology.task.install:The correct ceph version 19.0.0-2568.gafa73a9e is installed. 2024-04-01T13:29:41.986 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-01T13:29:41.986 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-01T13:29:41.986 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-01T13:29:42.019 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-01T13:29:42.019 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-01T13:29:42.052 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-01T13:29:42.052 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-01T13:29:42.052 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/daemon-helper 2024-04-01T13:29:42.089 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-01T13:29:42.159 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-01T13:29:42.159 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/daemon-helper 2024-04-01T13:29:42.186 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-01T13:29:42.252 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-01T13:29:42.253 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-01T13:29:42.253 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-01T13:29:42.283 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-01T13:29:42.352 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-01T13:29:42.353 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-01T13:29:42.379 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-01T13:29:42.446 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-01T13:29:42.446 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-01T13:29:42.446 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/stdin-killer 2024-04-01T13:29:42.475 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-01T13:29:42.545 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-01T13:29:42.545 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/stdin-killer 2024-04-01T13:29:42.571 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-01T13:29:42.638 INFO:teuthology.run_tasks:Running task ceph... 2024-04-01T13:29:42.719 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-01T13:29:42.719 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 777 /var/log/ceph 2024-04-01T13:29:42.722 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 777 /var/log/ceph 2024-04-01T13:29:42.748 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-01T13:29:42.748 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-01T13:29:42.792 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-01T13:29:42.818 INFO:tasks.ceph:Creating extra log directories... 2024-04-01T13:29:42.818 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-01T13:29:42.860 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-01T13:29:42.896 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-01T13:29:42.897 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-01T13:29:42.897 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-04-01_13:06:33-crimson-rados-wip-matanb-crimson-testing-thrash-options-distro-crimson-smithi/7633867', 'branch': 'wip-matanb-crimson-testing-thrash-options', 'description': 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects}', 'email': None, 'first_in_suite': False, 'job_id': '7633867', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-04-01_13:06:33-crimson-rados-wip-matanb-crimson-testing-thrash-options-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-testing-thrash-options'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'afa73a9e21b8fb0610f8af7266f1975be37c0a69'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'conf': {'osd': {'osd deep scrub update digest min age': 0}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'afa73a9e21b8fb0610f8af7266f1975be37c0a69'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-matanb-crimson-testing-thrash-options', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'afa73a9e21b8fb0610f8af7266f1975be37c0a69'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 5328, 'sha1': 'afa73a9e21b8fb0610f8af7266f1975be37c0a69', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-matanb-crimson-testing-thrash-options', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_afa73a9e21b8fb0610f8af7266f1975be37c0a69/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'afa73a9e21b8fb0610f8af7266f1975be37c0a69', 'targets': {'smithi059.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC2n7JhWy+x4pN0XflEn9IkPV0yMIou3ZCH758Jn0jgxXkH2bkeblRPAyOihpX2NzK8flS9Dj8L8m9L63z+f1n8=', 'smithi087.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMpS3tU09fadH186AmEtnwjQ3T8VRB9xlaUHIPkeQ8+YtDBUPptD4948Pb6nLVhx5gDd+ofjg3xzvhYmD5OK6vA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'afa73a9e21b8fb0610f8af7266f1975be37c0a69', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'rados': {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000, 'pool_snaps': True}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-04-01_13:06:33', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-01T13:29:42.897 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-01T13:29:42.929 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-01T13:29:42.944 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/run/ceph 2024-04-01T13:29:42.984 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/run/ceph 2024-04-01T13:29:43.011 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-01T13:29:43.011 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2024-04-01T13:29:43.066 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-01T13:29:43.066 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_1 2024-04-01T13:29:43.121 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-01T13:29:43.121 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T13:29:43.121 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-01T13:29:43.121 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T13:29:43.121 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T13:29:43.121 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-04-01 13:29:27.953786302 +0000 2024-04-01T13:29:43.121 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-04-01 13:26:11.698588070 +0000 2024-04-01T13:29:43.121 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-04-01 13:26:11.698588070 +0000 2024-04-01T13:29:43.121 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-04-01 13:26:11.698588070 +0000 2024-04-01T13:29:43.122 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-01T13:29:43.187 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-04-01T13:29:43.187 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-04-01T13:29:43.187 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000257224 s, 2.0 MB/s 2024-04-01T13:29:43.188 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-01T13:29:43.244 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_2 2024-04-01T13:29:43.298 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-01T13:29:43.299 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T13:29:43.299 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-01T13:29:43.299 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T13:29:43.299 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T13:29:43.299 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-04-01 13:29:27.954786289 +0000 2024-04-01T13:29:43.299 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-04-01 13:26:12.040583209 +0000 2024-04-01T13:29:43.299 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-04-01 13:26:12.040583209 +0000 2024-04-01T13:29:43.299 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-04-01 13:26:12.040583209 +0000 2024-04-01T13:29:43.299 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-01T13:29:43.364 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-04-01T13:29:43.364 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-04-01T13:29:43.364 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.00021389 s, 2.4 MB/s 2024-04-01T13:29:43.365 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-01T13:29:43.420 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_3 2024-04-01T13:29:43.476 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-01T13:29:43.476 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T13:29:43.476 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-01T13:29:43.476 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T13:29:43.476 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T13:29:43.476 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-04-01 13:29:27.954786289 +0000 2024-04-01T13:29:43.476 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-04-01 13:26:12.390578233 +0000 2024-04-01T13:29:43.476 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-04-01 13:26:12.390578233 +0000 2024-04-01T13:29:43.476 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-04-01 13:26:12.390578233 +0000 2024-04-01T13:29:43.477 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-01T13:29:43.541 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-04-01T13:29:43.542 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-04-01T13:29:43.542 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000168623 s, 3.0 MB/s 2024-04-01T13:29:43.543 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-01T13:29:43.598 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_4 2024-04-01T13:29:43.652 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-01T13:29:43.652 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T13:29:43.652 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-01T13:29:43.653 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T13:29:43.653 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T13:29:43.653 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-04-01 13:29:27.955786275 +0000 2024-04-01T13:29:43.653 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-04-01 13:26:12.749573130 +0000 2024-04-01T13:29:43.653 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-04-01 13:26:12.749573130 +0000 2024-04-01T13:29:43.653 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-04-01 13:26:12.749573130 +0000 2024-04-01T13:29:43.653 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-01T13:29:43.717 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-04-01T13:29:43.717 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-04-01T13:29:43.717 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000172567 s, 3.0 MB/s 2024-04-01T13:29:43.718 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-01T13:29:43.773 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-01T13:29:43.773 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-01T13:29:43.773 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2024-04-01T13:29:43.791 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-01T13:29:43.791 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_1 2024-04-01T13:29:43.847 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-01T13:29:43.847 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T13:29:43.847 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-01T13:29:43.847 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T13:29:43.848 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T13:29:43.848 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-04-01 13:29:37.953139353 +0000 2024-04-01T13:29:43.848 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-04-01 13:26:06.142252049 +0000 2024-04-01T13:29:43.848 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-04-01 13:26:06.142252049 +0000 2024-04-01T13:29:43.848 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-04-01 13:26:06.142252049 +0000 2024-04-01T13:29:43.848 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-01T13:29:43.914 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-04-01T13:29:43.914 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-04-01T13:29:43.914 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000189822 s, 2.7 MB/s 2024-04-01T13:29:43.915 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-01T13:29:43.971 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_2 2024-04-01T13:29:44.027 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-01T13:29:44.027 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T13:29:44.027 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-04-01T13:29:44.027 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T13:29:44.027 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T13:29:44.028 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-04-01 13:29:37.953139353 +0000 2024-04-01T13:29:44.028 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-04-01 13:26:06.494243625 +0000 2024-04-01T13:29:44.028 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-04-01 13:26:06.494243625 +0000 2024-04-01T13:29:44.028 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-04-01 13:26:06.494243625 +0000 2024-04-01T13:29:44.028 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-01T13:29:44.093 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-04-01T13:29:44.093 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-04-01T13:29:44.093 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000212389 s, 2.4 MB/s 2024-04-01T13:29:44.094 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-01T13:29:44.149 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_3 2024-04-01T13:29:44.205 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-01T13:29:44.205 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T13:29:44.205 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-01T13:29:44.205 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T13:29:44.205 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T13:29:44.205 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-04-01 13:29:37.954139329 +0000 2024-04-01T13:29:44.205 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-04-01 13:26:06.814235967 +0000 2024-04-01T13:29:44.205 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-04-01 13:26:06.814235967 +0000 2024-04-01T13:29:44.205 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-04-01 13:26:06.814235967 +0000 2024-04-01T13:29:44.206 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-01T13:29:44.270 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-04-01T13:29:44.270 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-04-01T13:29:44.270 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000207655 s, 2.5 MB/s 2024-04-01T13:29:44.271 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-01T13:29:44.326 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_4 2024-04-01T13:29:44.380 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-01T13:29:44.380 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T13:29:44.381 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-01T13:29:44.381 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T13:29:44.381 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T13:29:44.381 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-04-01 13:29:37.954139329 +0000 2024-04-01T13:29:44.381 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-04-01 13:26:07.154227830 +0000 2024-04-01T13:29:44.381 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-04-01 13:26:07.154227830 +0000 2024-04-01T13:29:44.381 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-04-01 13:26:07.154227830 +0000 2024-04-01T13:29:44.381 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-01T13:29:44.446 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-04-01T13:29:44.447 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-04-01T13:29:44.447 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000176096 s, 2.9 MB/s 2024-04-01T13:29:44.448 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-01T13:29:44.504 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-01T13:29:44.504 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-04-01T13:29:44.504 INFO:tasks.ceph:Generating config... 2024-04-01T13:29:44.506 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-01T13:29:44.506 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-01T13:29:44.506 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-01T13:29:44.506 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-01T13:29:44.506 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-01T13:29:44.506 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-04-01T13:29:44.506 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-01T13:29:44.506 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-01T13:29:44.506 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-01T13:29:44.506 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-01T13:29:44.507 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-01T13:29:44.507 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-01T13:29:44.507 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-01T13:29:44.507 INFO:tasks.ceph:[osd] err to stderr = True 2024-04-01T13:29:44.507 INFO:tasks.ceph:[osd] log flush on exit = True 2024-04-01T13:29:44.507 INFO:tasks.ceph:[osd] log to file = False 2024-04-01T13:29:44.507 INFO:tasks.ceph:[osd] log to stderr = True 2024-04-01T13:29:44.507 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-01T13:29:44.507 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-01T13:29:44.507 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-01T13:29:44.507 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-01T13:29:44.507 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-04-01T13:29:44.508 INFO:tasks.ceph:[osd] osd delete sleep = 1 2024-04-01T13:29:44.508 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-01T13:29:44.508 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-04-01T13:29:44.508 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-01T13:29:44.508 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-01T13:29:44.508 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-04-01T13:29:44.508 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-04-01T13:29:44.508 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-01T13:29:44.508 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-04-01T13:29:44.508 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-01T13:29:44.508 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-01T13:29:44.508 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-01T13:29:44.508 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-01T13:29:44.508 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-01T13:29:44.509 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-01T13:29:44.509 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-01T13:29:44.509 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-04-01T13:29:44.509 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2024-04-01T13:29:44.509 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-01T13:29:44.509 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-01T13:29:44.509 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-01T13:29:44.509 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-01T13:29:44.509 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-01T13:29:44.509 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-04-01T13:29:44.509 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-01T13:29:44.509 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-01T13:29:44.509 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-01T13:29:44.509 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-01T13:29:44.510 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-04-01T13:29:44.510 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-04-01T13:29:44.510 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-04-01T13:29:44.510 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-04-01T13:29:44.510 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-04-01T13:29:44.510 INFO:tasks.ceph:Setting up mon.a... 2024-04-01T13:29:44.510 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-01T13:29:44.583 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/ceph.keyring 2024-04-01T13:29:44.586 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-01T13:29:44.643 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-01T13:29:44.670 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.59')] 2024-04-01T13:29:44.670 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.59', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-04-01T13:29:44.671 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-01T13:29:44.671 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-01T13:29:44.725 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.59 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-01T13:29:44.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:29:44.830+0000 7fb230069f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:29:44.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:29:44.830+0000 7fb230069f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:29:44.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:29:44.830+0000 7fb230069f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:29:44.817 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-01T13:29:44.817 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: generated fsid 2a626e12-75d1-4fa9-b59b-fc64723c30fe 2024-04-01T13:29:44.817 INFO:teuthology.orchestra.run.smithi059.stdout:setting min_mon_release = quincy 2024-04-01T13:29:44.817 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 0 2024-04-01T13:29:44.817 INFO:teuthology.orchestra.run.smithi059.stdout:fsid 2a626e12-75d1-4fa9-b59b-fc64723c30fe 2024-04-01T13:29:44.818 INFO:teuthology.orchestra.run.smithi059.stdout:last_changed 2024-04-01T13:29:44.831770+0000 2024-04-01T13:29:44.818 INFO:teuthology.orchestra.run.smithi059.stdout:created 2024-04-01T13:29:44.831770+0000 2024-04-01T13:29:44.818 INFO:teuthology.orchestra.run.smithi059.stdout:min_mon_release 17 (quincy) 2024-04-01T13:29:44.818 INFO:teuthology.orchestra.run.smithi059.stdout:election_strategy: 1 2024-04-01T13:29:44.818 INFO:teuthology.orchestra.run.smithi059.stdout:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2024-04-01T13:29:44.818 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-01T13:29:44.819 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-01T13:29:44.833 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2a626e12-75d1-4fa9-b59b-fc64723c30fe... 2024-04-01T13:29:44.834 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-01T13:29:44.876 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-01T13:29:44.923 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2024-04-01T13:29:44.923 INFO:teuthology.orchestra.run.smithi059.stdout: chdir = "" 2024-04-01T13:29:44.923 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-01T13:29:44.923 INFO:teuthology.orchestra.run.smithi059.stdout: auth supported = cephx 2024-04-01T13:29:44.923 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.924 INFO:teuthology.orchestra.run.smithi059.stdout: filestore xattr use omap = true 2024-04-01T13:29:44.924 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.924 INFO:teuthology.orchestra.run.smithi059.stdout: mon clock drift allowed = 1.000 2024-04-01T13:29:44.924 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.924 INFO:teuthology.orchestra.run.smithi059.stdout: osd crush chooseleaf type = 0 2024-04-01T13:29:44.924 INFO:teuthology.orchestra.run.smithi059.stdout: auth debug = true 2024-04-01T13:29:44.924 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.924 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on old message = true 2024-04-01T13:29:44.924 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on bug = true 2024-04-01T13:29:44.924 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.924 INFO:teuthology.orchestra.run.smithi059.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-01T13:29:44.924 INFO:teuthology.orchestra.run.smithi059.stdout: mon pg warn max object skew = 0 2024-04-01T13:29:44.924 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.924 INFO:teuthology.orchestra.run.smithi059.stdout: # disable pg_autoscaler by default for new pools 2024-04-01T13:29:44.924 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-01T13:29:44.925 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.925 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default size = 2 2024-04-01T13:29:44.925 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.925 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow primary affinity = true 2024-04-01T13:29:44.925 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow pg remap = true 2024-04-01T13:29:44.925 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on legacy crush tunables = false 2024-04-01T13:29:44.925 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on crush straw calc version zero = false 2024-04-01T13:29:44.925 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on no sortbitwise = false 2024-04-01T13:29:44.925 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on osd down out interval zero = false 2024-04-01T13:29:44.925 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on too few osds = false 2024-04-01T13:29:44.925 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-01T13:29:44.925 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-01T13:29:44.925 INFO:teuthology.orchestra.run.smithi059.stdout: mon_allow_pool_size_one = true 2024-04-01T13:29:44.925 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.925 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-01T13:29:44.925 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.926 INFO:teuthology.orchestra.run.smithi059.stdout: osd default data pool replay window = 5 2024-04-01T13:29:44.926 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.926 INFO:teuthology.orchestra.run.smithi059.stdout: mon allow pool delete = true 2024-04-01T13:29:44.926 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.926 INFO:teuthology.orchestra.run.smithi059.stdout: mon cluster log file level = debug 2024-04-01T13:29:44.926 INFO:teuthology.orchestra.run.smithi059.stdout: debug asserts on shutdown = true 2024-04-01T13:29:44.926 INFO:teuthology.orchestra.run.smithi059.stdout: mon health detail to clog = false 2024-04-01T13:29:44.926 INFO:teuthology.orchestra.run.smithi059.stdout: mon host = 172.21.15.59 2024-04-01T13:29:44.926 INFO:teuthology.orchestra.run.smithi059.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-01T13:29:44.926 INFO:teuthology.orchestra.run.smithi059.stdout: ms client mode = crc 2024-04-01T13:29:44.926 INFO:teuthology.orchestra.run.smithi059.stdout: ms cluster mode = crc 2024-04-01T13:29:44.926 INFO:teuthology.orchestra.run.smithi059.stdout: ms mon client mode = crc 2024-04-01T13:29:44.926 INFO:teuthology.orchestra.run.smithi059.stdout: ms mon cluster mode = crc 2024-04-01T13:29:44.927 INFO:teuthology.orchestra.run.smithi059.stdout: ms mon service mode = crc 2024-04-01T13:29:44.927 INFO:teuthology.orchestra.run.smithi059.stdout: ms service mode = crc 2024-04-01T13:29:44.927 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_min_size = 2 2024-04-01T13:29:44.927 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_size = 3 2024-04-01T13:29:44.927 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = 2a626e12-75d1-4fa9-b59b-fc64723c30fe 2024-04-01T13:29:44.927 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.927 INFO:teuthology.orchestra.run.smithi059.stdout:[osd] 2024-04-01T13:29:44.927 INFO:teuthology.orchestra.run.smithi059.stdout: osd journal size = 100 2024-04-01T13:29:44.927 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.927 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub load threshold = 5.0 2024-04-01T13:29:44.927 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub max interval = 120 2024-04-01T13:29:44.927 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock profile = high_recovery_ops 2024-04-01T13:29:44.927 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.927 INFO:teuthology.orchestra.run.smithi059.stdout: osd recover clone overlap = true 2024-04-01T13:29:44.928 INFO:teuthology.orchestra.run.smithi059.stdout: osd recovery max chunk = 1048576 2024-04-01T13:29:44.928 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.928 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug shutdown = true 2024-04-01T13:29:44.928 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug op order = true 2024-04-01T13:29:44.928 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify stray on activate = true 2024-04-01T13:29:44.928 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug trim objects = true 2024-04-01T13:29:44.928 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.928 INFO:teuthology.orchestra.run.smithi059.stdout: osd open classes on start = true 2024-04-01T13:29:44.928 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug pg log writeout = true 2024-04-01T13:29:44.928 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.928 INFO:teuthology.orchestra.run.smithi059.stdout: osd deep scrub update digest min age = 30 2024-04-01T13:29:44.928 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.928 INFO:teuthology.orchestra.run.smithi059.stdout: osd map max advance = 10 2024-04-01T13:29:44.928 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.929 INFO:teuthology.orchestra.run.smithi059.stdout: journal zero on create = true 2024-04-01T13:29:44.929 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.929 INFO:teuthology.orchestra.run.smithi059.stdout: filestore ondisk finisher threads = 3 2024-04-01T13:29:44.929 INFO:teuthology.orchestra.run.smithi059.stdout: filestore apply finisher threads = 3 2024-04-01T13:29:44.929 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.929 INFO:teuthology.orchestra.run.smithi059.stdout: bdev debug aio = true 2024-04-01T13:29:44.929 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug misdirected ops = true 2024-04-01T13:29:44.929 INFO:teuthology.orchestra.run.smithi059.stdout: debug monc = 20 2024-04-01T13:29:44.929 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore block size = 96636764160 2024-04-01T13:29:44.929 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore compression algorithm = snappy 2024-04-01T13:29:44.929 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore compression mode = aggressive 2024-04-01T13:29:44.929 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore fsck on mount = True 2024-04-01T13:29:44.929 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore rocksdb cf = False 2024-04-01T13:29:44.930 INFO:teuthology.orchestra.run.smithi059.stdout: crimson alien thread cpu cores = 6-7 2024-04-01T13:29:44.930 INFO:teuthology.orchestra.run.smithi059.stdout: crimson osd obc lru size = 10 2024-04-01T13:29:44.930 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluefs = 20 2024-04-01T13:29:44.930 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluestore = 20 2024-04-01T13:29:44.930 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-04-01T13:29:44.930 INFO:teuthology.orchestra.run.smithi059.stdout: debug osd = 20 2024-04-01T13:29:44.930 INFO:teuthology.orchestra.run.smithi059.stdout: debug rocksdb = 10 2024-04-01T13:29:44.930 INFO:teuthology.orchestra.run.smithi059.stdout: err to stderr = True 2024-04-01T13:29:44.930 INFO:teuthology.orchestra.run.smithi059.stdout: log flush on exit = True 2024-04-01T13:29:44.930 INFO:teuthology.orchestra.run.smithi059.stdout: log to file = False 2024-04-01T13:29:44.930 INFO:teuthology.orchestra.run.smithi059.stdout: log to stderr = True 2024-04-01T13:29:44.930 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd backfillfull_ratio = 0.85 2024-04-01T13:29:44.930 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd full ratio = 0.9 2024-04-01T13:29:44.930 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd nearfull ratio = 0.8 2024-04-01T13:29:44.930 INFO:teuthology.orchestra.run.smithi059.stdout: osd blocked scrub grace period = 3600 2024-04-01T13:29:44.930 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug reject backfill probability = 0.3 2024-04-01T13:29:44.931 INFO:teuthology.orchestra.run.smithi059.stdout: osd delete sleep = 1 2024-04-01T13:29:44.931 INFO:teuthology.orchestra.run.smithi059.stdout: osd failsafe full ratio = 0.95 2024-04-01T13:29:44.931 INFO:teuthology.orchestra.run.smithi059.stdout: osd max backfills = 3 2024-04-01T13:29:44.931 INFO:teuthology.orchestra.run.smithi059.stdout: osd max markdown count = 1000 2024-04-01T13:29:44.931 INFO:teuthology.orchestra.run.smithi059.stdout: osd objectstore = bluestore 2024-04-01T13:29:44.931 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub min interval = 60 2024-04-01T13:29:44.931 INFO:teuthology.orchestra.run.smithi059.stdout: osd shutdown pgref assert = True 2024-04-01T13:29:44.931 INFO:teuthology.orchestra.run.smithi059.stdout: osd snap trim sleep = 2 2024-04-01T13:29:44.931 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.931 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr] 2024-04-01T13:29:44.931 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-04-01T13:29:44.931 INFO:teuthology.orchestra.run.smithi059.stdout: debug mgr = 20 2024-04-01T13:29:44.931 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-04-01T13:29:44.932 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-04-01T13:29:44.932 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-04-01T13:29:44.932 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-04-01T13:29:44.932 INFO:teuthology.orchestra.run.smithi059.stdout: mgr/telemetry/nag = false 2024-04-01T13:29:44.932 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.932 INFO:teuthology.orchestra.run.smithi059.stdout:[mon] 2024-04-01T13:29:44.932 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-04-01T13:29:44.932 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-04-01T13:29:44.932 INFO:teuthology.orchestra.run.smithi059.stdout: debug paxos = 20 2024-04-01T13:29:44.932 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-04-01T13:29:44.932 INFO:teuthology.orchestra.run.smithi059.stdout: mon data avail warn = 5 2024-04-01T13:29:44.932 INFO:teuthology.orchestra.run.smithi059.stdout: mon mgr mkfs grace = 240 2024-04-01T13:29:44.932 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-04-01T13:29:44.933 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd reporter subtree level = osd 2024-04-01T13:29:44.933 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd prime pg temp = true 2024-04-01T13:29:44.933 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-04-01T13:29:44.933 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.933 INFO:teuthology.orchestra.run.smithi059.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-01T13:29:44.933 INFO:teuthology.orchestra.run.smithi059.stdout: auth mon ticket ttl = 660 # 11m 2024-04-01T13:29:44.933 INFO:teuthology.orchestra.run.smithi059.stdout: auth service ticket ttl = 240 # 4m 2024-04-01T13:29:44.933 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.933 INFO:teuthology.orchestra.run.smithi059.stdout: # don't complain about insecure global_id in the test suite 2024-04-01T13:29:44.933 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-01T13:29:44.933 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-01T13:29:44.933 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.933 INFO:teuthology.orchestra.run.smithi059.stdout: # 1m isn't quite enough 2024-04-01T13:29:44.933 INFO:teuthology.orchestra.run.smithi059.stdout: mon_down_mkfs_grace = 2m 2024-04-01T13:29:44.934 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.934 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_filestore_osds = false 2024-04-01T13:29:44.934 INFO:teuthology.orchestra.run.smithi059.stdout: mon min osdmap epochs = 50 2024-04-01T13:29:44.934 INFO:teuthology.orchestra.run.smithi059.stdout: mon osdmap full prune interval = 2 2024-04-01T13:29:44.934 INFO:teuthology.orchestra.run.smithi059.stdout: mon osdmap full prune min = 15 2024-04-01T13:29:44.934 INFO:teuthology.orchestra.run.smithi059.stdout: mon osdmap full prune txsize = 2 2024-04-01T13:29:44.934 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default crimson = True 2024-04-01T13:29:44.934 INFO:teuthology.orchestra.run.smithi059.stdout: paxos service trim min = 10 2024-04-01T13:29:44.934 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:29:44.934 INFO:teuthology.orchestra.run.smithi059.stdout:[client] 2024-04-01T13:29:44.934 INFO:teuthology.orchestra.run.smithi059.stdout: rgw cache enabled = true 2024-04-01T13:29:44.934 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable ops log = true 2024-04-01T13:29:44.934 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable usage log = true 2024-04-01T13:29:44.934 INFO:teuthology.orchestra.run.smithi059.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-01T13:29:44.934 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-01T13:29:44.934 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.a] 2024-04-01T13:29:44.935 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.0] 2024-04-01T13:29:44.935 INFO:teuthology.orchestra.run.smithi059.stdout: crimson seastar cpu cores = 0-2 2024-04-01T13:29:44.935 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.1] 2024-04-01T13:29:44.935 INFO:teuthology.orchestra.run.smithi059.stdout: crimson seastar cpu cores = 3-5 2024-04-01T13:29:44.935 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.2] 2024-04-01T13:29:44.935 INFO:teuthology.orchestra.run.smithi059.stdout: crimson seastar cpu cores = 0-2 2024-04-01T13:29:44.935 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.3] 2024-04-01T13:29:44.935 INFO:teuthology.orchestra.run.smithi059.stdout: crimson seastar cpu cores = 3-5 2024-04-01T13:29:44.935 INFO:teuthology.orchestra.run.smithi087.stdout:[global] 2024-04-01T13:29:44.935 INFO:teuthology.orchestra.run.smithi087.stdout: chdir = "" 2024-04-01T13:29:44.935 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-01T13:29:44.936 INFO:teuthology.orchestra.run.smithi087.stdout: auth supported = cephx 2024-04-01T13:29:44.936 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.936 INFO:teuthology.orchestra.run.smithi087.stdout: filestore xattr use omap = true 2024-04-01T13:29:44.936 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.936 INFO:teuthology.orchestra.run.smithi087.stdout: mon clock drift allowed = 1.000 2024-04-01T13:29:44.936 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.936 INFO:teuthology.orchestra.run.smithi087.stdout: osd crush chooseleaf type = 0 2024-04-01T13:29:44.936 INFO:teuthology.orchestra.run.smithi087.stdout: auth debug = true 2024-04-01T13:29:44.936 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.936 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on old message = true 2024-04-01T13:29:44.936 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on bug = true 2024-04-01T13:29:44.936 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.936 INFO:teuthology.orchestra.run.smithi087.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-01T13:29:44.936 INFO:teuthology.orchestra.run.smithi087.stdout: mon pg warn max object skew = 0 2024-04-01T13:29:44.936 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.936 INFO:teuthology.orchestra.run.smithi087.stdout: # disable pg_autoscaler by default for new pools 2024-04-01T13:29:44.937 INFO:teuthology.orchestra.run.smithi087.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-01T13:29:44.937 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.937 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default size = 2 2024-04-01T13:29:44.937 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.937 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow primary affinity = true 2024-04-01T13:29:44.937 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow pg remap = true 2024-04-01T13:29:44.937 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on legacy crush tunables = false 2024-04-01T13:29:44.937 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on crush straw calc version zero = false 2024-04-01T13:29:44.937 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on no sortbitwise = false 2024-04-01T13:29:44.937 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on osd down out interval zero = false 2024-04-01T13:29:44.937 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on too few osds = false 2024-04-01T13:29:44.937 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-01T13:29:44.937 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-01T13:29:44.937 INFO:teuthology.orchestra.run.smithi087.stdout: mon_allow_pool_size_one = true 2024-04-01T13:29:44.937 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.937 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-01T13:29:44.938 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.938 INFO:teuthology.orchestra.run.smithi087.stdout: osd default data pool replay window = 5 2024-04-01T13:29:44.938 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.938 INFO:teuthology.orchestra.run.smithi087.stdout: mon allow pool delete = true 2024-04-01T13:29:44.938 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.938 INFO:teuthology.orchestra.run.smithi087.stdout: mon cluster log file level = debug 2024-04-01T13:29:44.938 INFO:teuthology.orchestra.run.smithi087.stdout: debug asserts on shutdown = true 2024-04-01T13:29:44.938 INFO:teuthology.orchestra.run.smithi087.stdout: mon health detail to clog = false 2024-04-01T13:29:44.938 INFO:teuthology.orchestra.run.smithi087.stdout: mon host = 172.21.15.59 2024-04-01T13:29:44.938 INFO:teuthology.orchestra.run.smithi087.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-01T13:29:44.938 INFO:teuthology.orchestra.run.smithi087.stdout: ms client mode = crc 2024-04-01T13:29:44.938 INFO:teuthology.orchestra.run.smithi087.stdout: ms cluster mode = crc 2024-04-01T13:29:44.938 INFO:teuthology.orchestra.run.smithi087.stdout: ms mon client mode = crc 2024-04-01T13:29:44.938 INFO:teuthology.orchestra.run.smithi087.stdout: ms mon cluster mode = crc 2024-04-01T13:29:44.938 INFO:teuthology.orchestra.run.smithi087.stdout: ms mon service mode = crc 2024-04-01T13:29:44.938 INFO:teuthology.orchestra.run.smithi087.stdout: ms service mode = crc 2024-04-01T13:29:44.939 INFO:teuthology.orchestra.run.smithi087.stdout: osd_pool_default_min_size = 2 2024-04-01T13:29:44.939 INFO:teuthology.orchestra.run.smithi087.stdout: osd_pool_default_size = 3 2024-04-01T13:29:44.939 INFO:teuthology.orchestra.run.smithi087.stdout: fsid = 2a626e12-75d1-4fa9-b59b-fc64723c30fe 2024-04-01T13:29:44.939 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.939 INFO:teuthology.orchestra.run.smithi087.stdout:[osd] 2024-04-01T13:29:44.939 INFO:teuthology.orchestra.run.smithi087.stdout: osd journal size = 100 2024-04-01T13:29:44.939 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.939 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub load threshold = 5.0 2024-04-01T13:29:44.939 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub max interval = 120 2024-04-01T13:29:44.939 INFO:teuthology.orchestra.run.smithi087.stdout: osd mclock profile = high_recovery_ops 2024-04-01T13:29:44.939 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.939 INFO:teuthology.orchestra.run.smithi087.stdout: osd recover clone overlap = true 2024-04-01T13:29:44.939 INFO:teuthology.orchestra.run.smithi087.stdout: osd recovery max chunk = 1048576 2024-04-01T13:29:44.939 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.939 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug shutdown = true 2024-04-01T13:29:44.939 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug op order = true 2024-04-01T13:29:44.940 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify stray on activate = true 2024-04-01T13:29:44.940 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug trim objects = true 2024-04-01T13:29:44.940 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.940 INFO:teuthology.orchestra.run.smithi087.stdout: osd open classes on start = true 2024-04-01T13:29:44.940 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug pg log writeout = true 2024-04-01T13:29:44.940 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.940 INFO:teuthology.orchestra.run.smithi087.stdout: osd deep scrub update digest min age = 30 2024-04-01T13:29:44.940 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.940 INFO:teuthology.orchestra.run.smithi087.stdout: osd map max advance = 10 2024-04-01T13:29:44.940 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.940 INFO:teuthology.orchestra.run.smithi087.stdout: journal zero on create = true 2024-04-01T13:29:44.940 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.940 INFO:teuthology.orchestra.run.smithi087.stdout: filestore ondisk finisher threads = 3 2024-04-01T13:29:44.940 INFO:teuthology.orchestra.run.smithi087.stdout: filestore apply finisher threads = 3 2024-04-01T13:29:44.940 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.940 INFO:teuthology.orchestra.run.smithi087.stdout: bdev debug aio = true 2024-04-01T13:29:44.941 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug misdirected ops = true 2024-04-01T13:29:44.941 INFO:teuthology.orchestra.run.smithi087.stdout: debug monc = 20 2024-04-01T13:29:44.941 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore block size = 96636764160 2024-04-01T13:29:44.941 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore compression algorithm = snappy 2024-04-01T13:29:44.941 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore compression mode = aggressive 2024-04-01T13:29:44.941 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore fsck on mount = True 2024-04-01T13:29:44.941 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore rocksdb cf = False 2024-04-01T13:29:44.941 INFO:teuthology.orchestra.run.smithi087.stdout: crimson alien thread cpu cores = 6-7 2024-04-01T13:29:44.941 INFO:teuthology.orchestra.run.smithi087.stdout: crimson osd obc lru size = 10 2024-04-01T13:29:44.941 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluefs = 20 2024-04-01T13:29:44.941 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluestore = 20 2024-04-01T13:29:44.941 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-04-01T13:29:44.941 INFO:teuthology.orchestra.run.smithi087.stdout: debug osd = 20 2024-04-01T13:29:44.941 INFO:teuthology.orchestra.run.smithi087.stdout: debug rocksdb = 10 2024-04-01T13:29:44.942 INFO:teuthology.orchestra.run.smithi087.stdout: err to stderr = True 2024-04-01T13:29:44.942 INFO:teuthology.orchestra.run.smithi087.stdout: log flush on exit = True 2024-04-01T13:29:44.942 INFO:teuthology.orchestra.run.smithi087.stdout: log to file = False 2024-04-01T13:29:44.942 INFO:teuthology.orchestra.run.smithi087.stdout: log to stderr = True 2024-04-01T13:29:44.942 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd backfillfull_ratio = 0.85 2024-04-01T13:29:44.942 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd full ratio = 0.9 2024-04-01T13:29:44.942 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd nearfull ratio = 0.8 2024-04-01T13:29:44.942 INFO:teuthology.orchestra.run.smithi087.stdout: osd blocked scrub grace period = 3600 2024-04-01T13:29:44.942 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug reject backfill probability = 0.3 2024-04-01T13:29:44.942 INFO:teuthology.orchestra.run.smithi087.stdout: osd delete sleep = 1 2024-04-01T13:29:44.942 INFO:teuthology.orchestra.run.smithi087.stdout: osd failsafe full ratio = 0.95 2024-04-01T13:29:44.942 INFO:teuthology.orchestra.run.smithi087.stdout: osd max backfills = 3 2024-04-01T13:29:44.942 INFO:teuthology.orchestra.run.smithi087.stdout: osd max markdown count = 1000 2024-04-01T13:29:44.942 INFO:teuthology.orchestra.run.smithi087.stdout: osd objectstore = bluestore 2024-04-01T13:29:44.943 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub min interval = 60 2024-04-01T13:29:44.943 INFO:teuthology.orchestra.run.smithi087.stdout: osd shutdown pgref assert = True 2024-04-01T13:29:44.943 INFO:teuthology.orchestra.run.smithi087.stdout: osd snap trim sleep = 2 2024-04-01T13:29:44.943 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.943 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr] 2024-04-01T13:29:44.943 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-04-01T13:29:44.943 INFO:teuthology.orchestra.run.smithi087.stdout: debug mgr = 20 2024-04-01T13:29:44.943 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2024-04-01T13:29:44.943 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2024-04-01T13:29:44.943 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2024-04-01T13:29:44.943 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2024-04-01T13:29:44.943 INFO:teuthology.orchestra.run.smithi087.stdout: mgr/telemetry/nag = false 2024-04-01T13:29:44.943 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.943 INFO:teuthology.orchestra.run.smithi087.stdout:[mon] 2024-04-01T13:29:44.943 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-04-01T13:29:44.944 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2024-04-01T13:29:44.944 INFO:teuthology.orchestra.run.smithi087.stdout: debug paxos = 20 2024-04-01T13:29:44.944 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2024-04-01T13:29:44.944 INFO:teuthology.orchestra.run.smithi087.stdout: mon data avail warn = 5 2024-04-01T13:29:44.944 INFO:teuthology.orchestra.run.smithi087.stdout: mon mgr mkfs grace = 240 2024-04-01T13:29:44.944 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2024-04-01T13:29:44.944 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd reporter subtree level = osd 2024-04-01T13:29:44.944 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd prime pg temp = true 2024-04-01T13:29:44.944 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2024-04-01T13:29:44.944 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.944 INFO:teuthology.orchestra.run.smithi087.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-01T13:29:44.944 INFO:teuthology.orchestra.run.smithi087.stdout: auth mon ticket ttl = 660 # 11m 2024-04-01T13:29:44.944 INFO:teuthology.orchestra.run.smithi087.stdout: auth service ticket ttl = 240 # 4m 2024-04-01T13:29:44.944 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.944 INFO:teuthology.orchestra.run.smithi087.stdout: # don't complain about insecure global_id in the test suite 2024-04-01T13:29:44.944 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-01T13:29:44.945 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-01T13:29:44.945 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.945 INFO:teuthology.orchestra.run.smithi087.stdout: # 1m isn't quite enough 2024-04-01T13:29:44.945 INFO:teuthology.orchestra.run.smithi087.stdout: mon_down_mkfs_grace = 2m 2024-04-01T13:29:44.945 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.945 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_filestore_osds = false 2024-04-01T13:29:44.945 INFO:teuthology.orchestra.run.smithi087.stdout: mon min osdmap epochs = 50 2024-04-01T13:29:44.945 INFO:teuthology.orchestra.run.smithi087.stdout: mon osdmap full prune interval = 2 2024-04-01T13:29:44.945 INFO:teuthology.orchestra.run.smithi087.stdout: mon osdmap full prune min = 15 2024-04-01T13:29:44.945 INFO:teuthology.orchestra.run.smithi087.stdout: mon osdmap full prune txsize = 2 2024-04-01T13:29:44.945 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default crimson = True 2024-04-01T13:29:44.945 INFO:teuthology.orchestra.run.smithi087.stdout: paxos service trim min = 10 2024-04-01T13:29:44.945 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-01T13:29:44.946 INFO:teuthology.orchestra.run.smithi087.stdout:[client] 2024-04-01T13:29:44.946 INFO:teuthology.orchestra.run.smithi087.stdout: rgw cache enabled = true 2024-04-01T13:29:44.946 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable ops log = true 2024-04-01T13:29:44.946 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable usage log = true 2024-04-01T13:29:44.946 INFO:teuthology.orchestra.run.smithi087.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-01T13:29:44.946 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-01T13:29:44.946 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.a] 2024-04-01T13:29:44.946 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.0] 2024-04-01T13:29:44.946 INFO:teuthology.orchestra.run.smithi087.stdout: crimson seastar cpu cores = 0-2 2024-04-01T13:29:44.946 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.1] 2024-04-01T13:29:44.946 INFO:teuthology.orchestra.run.smithi087.stdout: crimson seastar cpu cores = 3-5 2024-04-01T13:29:44.946 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.2] 2024-04-01T13:29:44.946 INFO:teuthology.orchestra.run.smithi087.stdout: crimson seastar cpu cores = 0-2 2024-04-01T13:29:44.946 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.3] 2024-04-01T13:29:44.946 INFO:teuthology.orchestra.run.smithi087.stdout: crimson seastar cpu cores = 3-5 2024-04-01T13:29:44.948 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-01T13:29:44.948 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-01T13:29:45.014 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-01T13:29:45.014 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-01T13:29:45.014 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-01T13:29:45.028 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-01T13:29:45.029 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-01T13:29:45.083 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi059.front.sepia.ceph.com 2024-04-01T13:29:45.083 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-01T13:29:45.083 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-01T13:29:45.084 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-01T13:29:45.161 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-01T13:29:45.161 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-01T13:29:45.217 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi087.front.sepia.ceph.com 2024-04-01T13:29:45.217 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-01T13:29:45.218 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-01T13:29:45.218 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-01T13:29:45.258 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-01T13:29:45.258 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-01T13:29:45.314 INFO:tasks.ceph:Setting up mon nodes... 2024-04-01T13:29:45.314 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-01T13:29:45.314 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-01T13:29:45.425 INFO:teuthology.orchestra.run.smithi087.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-01T13:29:45.428 INFO:tasks.ceph:Setting up mds nodes... 2024-04-01T13:29:45.428 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-01T13:29:45.428 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-01T13:29:45.479 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-01T13:29:45.494 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-01T13:29:45.546 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-01T13:29:45.562 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-01T13:29:45.562 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-04-01T13:29:45.562 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-01T13:29:45.588 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-01T13:29:45.589 INFO:tasks.ceph:role: osd.0 2024-04-01T13:29:45.589 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com 2024-04-01T13:29:45.589 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-01T13:29:45.663 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-01T13:29:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-01T13:29:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-01T13:29:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-01T13:29:45.663 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-01T13:29:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-04-01T13:29:45.663 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-01T13:29:45.663 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-01T13:29:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-01T13:29:45.663 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-01T13:29:46.178 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-04-01T13:29:46.180 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-04-01T13:29:46.180 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-01T13:29:46.320 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-01T13:29:46.354 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-01T13:29:46.421 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-01T13:29:46.422 INFO:tasks.ceph:role: osd.1 2024-04-01T13:29:46.422 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com 2024-04-01T13:29:46.422 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-01T13:29:46.496 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-01T13:29:46.496 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-01T13:29:46.496 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-01T13:29:46.496 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-01T13:29:46.497 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-01T13:29:46.497 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-04-01T13:29:46.497 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-01T13:29:46.497 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-01T13:29:46.497 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-01T13:29:46.497 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-01T13:29:46.940 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-04-01T13:29:46.942 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-04-01T13:29:46.942 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-01T13:29:47.026 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-01T13:29:47.098 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-01T13:29:47.420 INFO:teuthology.orchestra.run.smithi059.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-01T13:29:47.421 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-04-01 13:29:47,435 seastar - Reactor backend: linux-aio 2024-04-01T13:29:47.436 INFO:teuthology.orchestra.run.smithi059.stderr:==28813==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-01T13:29:47.543 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9a0449 0xb872f10 0xb87448c 0xb8746a2 0xb8747f8 0xb874cc1 0x54daf 0x362fa 0x5db4814 0x5db4a2a 0x5db4c0d 0x5db4ee3 0x5e03723 0x5e03929 0x5d82410 0x5d825ea 0x5d82b76 0x5bc792b 0x5be29ef 0x6ebcfa4 0x6ece777 0x6ecfc14 0x6ecfc99 0x6ecfd9b 0x6ecfe93 0x6ed00a0 0x6ed0286 0x6ed05d5 0x6ed0979 0x6ed0ba9 0x6ed0e13 0x6ed10e0 0x6ed12fe 0x6ed13b6 0x6ed147c 0x6ed155d 0x5a2df32 0x5a2e03c 0x5a2e0f8 0x5a2e1b4 0x5a2e270 0x5a2e3bd 0x5a2e548 0x5a2e909 0x5a4b386 0xb885b95 0xb89fe2f 0xb940e2d 0xb94247b 0xb61ebe3 0xb61f55f 0x37e0266 0x37e0dbf 0x3680d7a 0x3feaf 0x3ff5f 0x346d444 2024-04-01T13:29:47.543 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-04-01T13:29:47.569 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-04-01 13:29:47,583 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-01T13:29:47.569 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-04-01 13:29:47,584 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-04-01T13:29:47.570 INFO:teuthology.orchestra.run.smithi059.stderr:DEBUG 2024-04-01 13:29:47,584 [shard 0:main] seastar - reactor::drain 2024-04-01T13:29:48.152 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-04-01T13:29:48.152 INFO:teuthology.orchestra.run.smithi059.stderr:Suppressions used: 2024-04-01T13:29:48.152 INFO:teuthology.orchestra.run.smithi059.stderr: count bytes template 2024-04-01T13:29:48.152 INFO:teuthology.orchestra.run.smithi059.stderr: 1 8 MallocExtension::Register 2024-04-01T13:29:48.152 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-04-01T13:29:48.152 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-04-01T13:29:48.211 INFO:teuthology.orchestra.run.smithi059.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-01T13:29:48.214 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-04-01 13:29:48,229 seastar - Reactor backend: linux-aio 2024-04-01T13:29:48.223 INFO:teuthology.orchestra.run.smithi059.stderr:==28811==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-01T13:29:48.288 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9a0449 0xb872f10 0xb87448c 0xb8746a2 0xb8747f8 0xb874cc1 0x54daf 0x4a073 0x35b7c 0x3689e 0x3697876 0x5da7727 0x5da7911 0x5e03894 0x5d82410 0x5d825ea 0x5d82b76 0x5bc792b 0x5be29ef 0x6ebcfa4 0x6ece777 0x6ecfc14 0x6ecfc99 0x6ecfd9b 0x6ecfe93 0x6ed00a0 0x6ed0286 0x6ed05d5 0x6ed0979 0x6ed0ba9 0x6ed0e13 0x6ed10e0 0x6ed12fe 0x6ed13b6 0x6ed147c 0x6ed155d 0x5a2df32 0x5a2e03c 0x5a2e0f8 0x5a2e1b4 0x5a2e270 0x5a2e3bd 0x5a2e548 0x5a2e909 0x5a4b386 0xb885b95 0xb89fe2f 0xb940e2d 0xb94247b 0xb61ebe3 0xb61f55f 0x368158a 0x3feaf 0x3ff5f 0x346d444 2024-04-01T13:29:48.288 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-04-01T13:29:52.910 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-04-01T13:29:52.910 INFO:teuthology.orchestra.run.smithi059.stderr:Suppressions used: 2024-04-01T13:29:52.910 INFO:teuthology.orchestra.run.smithi059.stderr: count bytes template 2024-04-01T13:29:52.910 INFO:teuthology.orchestra.run.smithi059.stderr: 1 8 MallocExtension::Register 2024-04-01T13:29:52.910 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-04-01T13:29:52.910 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-04-01T13:29:52.911 INFO:teuthology.orchestra.run.smithi059.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 2a626e12-75d1-4fa9-b59b-fc64723c30fe 2024-04-01T13:29:52.933 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-01T13:29:52.960 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-01T13:29:53.250 INFO:teuthology.orchestra.run.smithi059.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-01T13:29:53.251 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-04-01 13:29:53,265 seastar - Reactor backend: linux-aio 2024-04-01T13:29:53.256 INFO:teuthology.orchestra.run.smithi059.stderr:==28996==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-01T13:29:53.363 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9a0449 0xb872f10 0xb87448c 0xb8746a2 0xb8747f8 0xb874cc1 0x54daf 0x174af 0x2eab 0x4a09d 0x35b7c 0x3689e 0x3697876 0x5da7727 0x5da7911 0x5e03894 0x5d82410 0x5d82944 0x5bc792b 0x5be29ef 0x6ebcfa4 0x6ece777 0x6ecfc14 0x6ecfc99 0x6ecfd9b 0x6ecfe93 0x6ed00a0 0x6ed0286 0x6ed05d5 0x6ed0979 0x6ed0ba9 0x6ed0e13 0x6ed10e0 0x6ed12fe 0x6ed13b6 0x6ed147c 0x6ed155d 0x5a2df32 0x5a2e03c 0x5a2e0f8 0x5a2e1b4 0x5a2e270 0x5a2e3bd 0x5a2e548 0x5a2e909 0x5a4b386 0xb885b95 0xb89fe2f 0xb940e2d 0xb94247b 0xb61ebe3 0xb61f55f 0x37e0266 0x37e0dbf 0x3680d7a 0x3feaf 0x3ff5f 0x346d444 2024-04-01T13:29:53.363 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-04-01T13:29:53.389 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-04-01 13:29:53,404 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-01T13:29:53.389 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-04-01 13:29:53,404 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-04-01T13:29:53.390 INFO:teuthology.orchestra.run.smithi059.stderr:DEBUG 2024-04-01 13:29:53,405 [shard 0:main] seastar - reactor::drain 2024-04-01T13:29:53.940 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-04-01T13:29:53.940 INFO:teuthology.orchestra.run.smithi059.stderr:Suppressions used: 2024-04-01T13:29:53.941 INFO:teuthology.orchestra.run.smithi059.stderr: count bytes template 2024-04-01T13:29:53.941 INFO:teuthology.orchestra.run.smithi059.stderr: 1 8 MallocExtension::Register 2024-04-01T13:29:53.941 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-04-01T13:29:53.941 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-04-01T13:29:54.010 INFO:teuthology.orchestra.run.smithi059.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-01T13:29:54.014 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-04-01 13:29:54,028 seastar - Reactor backend: linux-aio 2024-04-01T13:29:54.022 INFO:teuthology.orchestra.run.smithi059.stderr:==28994==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-01T13:29:54.087 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9a0449 0xb872f10 0xb87448c 0xb8746a2 0xb8747f8 0xb874cc1 0x54daf 0x17456 0x174bb 0x2eab 0x4a09d 0x35b7c 0x3629e 0x37e9ce2 0x43217f4 0x43c7fb8 0x5da6581 0x5d829c5 0x5bc792b 0x5be29ef 0x6ebcfa4 0x6ece777 0x6ecfc14 0x6ecfc99 0x6ecfd9b 0x6ecfe93 0x6ed00a0 0x6ed0286 0x6ed05d5 0x6ed0979 0x6ed0ba9 0x6ed0e13 0x6ed10e0 0x6ed12fe 0x6ed13b6 0x6ed147c 0x6ed155d 0x5a2df32 0x5a2e03c 0x5a2e0f8 0x5a2e1b4 0x5a2e270 0x5a2e3bd 0x5a2e548 0x5a2e909 0x5a4b386 0xb885b95 0xb89fe2f 0xb940e2d 0xb94247b 0xb61ebe3 0xb61f55f 0x368158a 0x3feaf 0x3ff5f 0x346d444 2024-04-01T13:29:54.088 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-04-01T13:29:58.701 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-04-01T13:29:58.701 INFO:teuthology.orchestra.run.smithi059.stderr:Suppressions used: 2024-04-01T13:29:58.701 INFO:teuthology.orchestra.run.smithi059.stderr: count bytes template 2024-04-01T13:29:58.701 INFO:teuthology.orchestra.run.smithi059.stderr: 1 8 MallocExtension::Register 2024-04-01T13:29:58.701 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-04-01T13:29:58.701 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-04-01T13:29:58.701 INFO:teuthology.orchestra.run.smithi059.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 2a626e12-75d1-4fa9-b59b-fc64723c30fe 2024-04-01T13:29:58.722 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-01T13:29:58.749 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-01T13:29:58.781 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-01T13:29:58.781 INFO:tasks.ceph:role: osd.2 2024-04-01T13:29:58.781 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com 2024-04-01T13:29:58.781 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-01T13:29:58.859 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-01T13:29:58.859 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-01T13:29:58.859 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-01T13:29:58.859 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-01T13:29:58.859 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-01T13:29:58.859 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-04-01T13:29:58.859 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-01T13:29:58.859 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-01T13:29:58.859 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-01T13:29:58.860 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-01T13:29:59.379 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-04-01T13:29:59.381 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-04-01T13:29:59.381 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-04-01T13:29:59.528 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-01T13:29:59.567 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-01T13:29:59.637 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-01T13:29:59.637 INFO:tasks.ceph:role: osd.3 2024-04-01T13:29:59.637 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com 2024-04-01T13:29:59.637 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-01T13:29:59.713 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-01T13:29:59.713 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-01T13:29:59.713 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-01T13:29:59.713 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-01T13:29:59.713 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-01T13:29:59.713 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-04-01T13:29:59.713 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-01T13:29:59.713 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-01T13:29:59.713 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-01T13:29:59.713 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-01T13:30:00.164 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-04-01T13:30:00.167 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-04-01T13:30:00.167 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-04-01T13:30:00.248 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-01T13:30:00.319 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-01T13:30:00.638 INFO:teuthology.orchestra.run.smithi087.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-01T13:30:00.639 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-04-01 13:30:00,654 seastar - Reactor backend: linux-aio 2024-04-01T13:30:00.658 INFO:teuthology.orchestra.run.smithi087.stderr:==29092==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-01T13:30:00.785 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-04-01 13:30:00,800 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-01T13:30:00.785 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-04-01 13:30:00,800 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-04-01T13:30:00.786 INFO:teuthology.orchestra.run.smithi087.stderr:DEBUG 2024-04-01 13:30:00,800 [shard 0:main] seastar - reactor::drain 2024-04-01T13:30:01.348 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-04-01T13:30:01.349 INFO:teuthology.orchestra.run.smithi087.stderr:Suppressions used: 2024-04-01T13:30:01.349 INFO:teuthology.orchestra.run.smithi087.stderr: count bytes template 2024-04-01T13:30:01.349 INFO:teuthology.orchestra.run.smithi087.stderr: 1 8 MallocExtension::Register 2024-04-01T13:30:01.349 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-04-01T13:30:01.349 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-04-01T13:30:01.407 INFO:teuthology.orchestra.run.smithi087.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-01T13:30:01.410 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-04-01 13:30:01,425 seastar - Reactor backend: linux-aio 2024-04-01T13:30:01.419 INFO:teuthology.orchestra.run.smithi087.stderr:==29090==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-01T13:30:06.060 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-04-01T13:30:06.061 INFO:teuthology.orchestra.run.smithi087.stderr:Suppressions used: 2024-04-01T13:30:06.061 INFO:teuthology.orchestra.run.smithi087.stderr: count bytes template 2024-04-01T13:30:06.061 INFO:teuthology.orchestra.run.smithi087.stderr: 1 8 MallocExtension::Register 2024-04-01T13:30:06.061 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-04-01T13:30:06.061 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-04-01T13:30:06.061 INFO:teuthology.orchestra.run.smithi087.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 2a626e12-75d1-4fa9-b59b-fc64723c30fe 2024-04-01T13:30:06.083 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-01T13:30:06.110 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-01T13:30:06.396 INFO:teuthology.orchestra.run.smithi087.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-01T13:30:06.397 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-04-01 13:30:06,412 seastar - Reactor backend: linux-aio 2024-04-01T13:30:06.402 INFO:teuthology.orchestra.run.smithi087.stderr:==29275==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-01T13:30:06.526 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-04-01 13:30:06,541 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-01T13:30:06.526 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-04-01 13:30:06,541 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-04-01T13:30:06.527 INFO:teuthology.orchestra.run.smithi087.stderr:DEBUG 2024-04-01 13:30:06,542 [shard 0:main] seastar - reactor::drain 2024-04-01T13:30:07.082 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-04-01T13:30:07.082 INFO:teuthology.orchestra.run.smithi087.stderr:Suppressions used: 2024-04-01T13:30:07.082 INFO:teuthology.orchestra.run.smithi087.stderr: count bytes template 2024-04-01T13:30:07.082 INFO:teuthology.orchestra.run.smithi087.stderr: 1 8 MallocExtension::Register 2024-04-01T13:30:07.082 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-04-01T13:30:07.082 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-04-01T13:30:07.144 INFO:teuthology.orchestra.run.smithi087.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-01T13:30:07.148 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-04-01 13:30:07,162 seastar - Reactor backend: linux-aio 2024-04-01T13:30:07.156 INFO:teuthology.orchestra.run.smithi087.stderr:==29273==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-01T13:30:11.797 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-04-01T13:30:11.797 INFO:teuthology.orchestra.run.smithi087.stderr:Suppressions used: 2024-04-01T13:30:11.797 INFO:teuthology.orchestra.run.smithi087.stderr: count bytes template 2024-04-01T13:30:11.797 INFO:teuthology.orchestra.run.smithi087.stderr: 1 8 MallocExtension::Register 2024-04-01T13:30:11.797 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-04-01T13:30:11.797 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-04-01T13:30:11.798 INFO:teuthology.orchestra.run.smithi087.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid 2a626e12-75d1-4fa9-b59b-fc64723c30fe 2024-04-01T13:30:11.819 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-01T13:30:11.845 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-01T13:30:11.846 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-01T13:30:11.846 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-01T13:30:11.878 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-01T13:30:11.878 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-01T13:30:11.948 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-01T13:30:11.948 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-01T13:30:11.975 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-01T13:30:11.975 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-01T13:30:12.042 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-01T13:30:12.042 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-01T13:30:12.109 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-01T13:30:12.109 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-01T13:30:12.126 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-01T13:30:12.126 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-01T13:30:12.164 INFO:tasks.ceph:Adding keys to all mons... 2024-04-01T13:30:12.164 DEBUG:teuthology.orchestra.run.smithi059:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-01T13:30:12.194 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.0] 2024-04-01T13:30:12.195 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQDMtgpm8gOaFxAA1gK2V5UrnuFNQnTL3TG7gA== 2024-04-01T13:30:12.195 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.1] 2024-04-01T13:30:12.195 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQDStgpmED2UCxAAk+4ImvT/yML9vHe8RlKi5w== 2024-04-01T13:30:12.195 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr.x] 2024-04-01T13:30:12.195 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQDJtgpmgzc/GhAAtJ9ae0HDUr2S154bRebTwQ== 2024-04-01T13:30:12.195 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.2] 2024-04-01T13:30:12.195 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQDZtgpmghOSIhAAjHF4pI63V5GEdTv0RCH7aw== 2024-04-01T13:30:12.195 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.3] 2024-04-01T13:30:12.195 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQDftgpmXR/5EhAAzh40b3SfEhXhkIYK/507Qg== 2024-04-01T13:30:12.195 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-04-01T13:30:12.195 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQDJtgpm2QxxHRAAurgJrAIQC+rQQZNXasxQ2g== 2024-04-01T13:30:12.195 INFO:teuthology.orchestra.run.smithi059.stdout:[client.1] 2024-04-01T13:30:12.195 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQDJtgpmrhl1IRAA/+K0gJzJn73TO+YylzNHbQ== 2024-04-01T13:30:12.196 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-01T13:30:12.306 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-01T13:30:12.370 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-01T13:30:12.434 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-01T13:30:12.494 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-01T13:30:12.555 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-01T13:30:12.620 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-01T13:30:12.685 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-01T13:30:12.685 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-01T13:30:12.711 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-01T13:30:12.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:12.831+0000 7fa17724ea80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:12.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:12.831+0000 7fa17724ea80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:12.831 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-01T13:30:12.857 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-01T13:30:12.911 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-01T13:30:12.911 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-01T13:30:12.912 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-01T13:30:12.953 INFO:tasks.ceph.mon.a:Started 2024-04-01T13:30:12.953 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-01T13:30:12.953 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-01T13:30:12.954 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-01T13:30:12.956 INFO:tasks.ceph.mgr.x:Started 2024-04-01T13:30:12.956 INFO:tasks.ceph:Setting crush tunables to default 2024-04-01T13:30:12.956 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd crush tunables default 2024-04-01T13:30:13.036 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:30:13.050+0000 7f797d862a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:13.036 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:30:13.050+0000 7f797d862a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:13.036 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:30:13.050+0000 7f797d862a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:13.053 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:13.066+0000 7fb506769000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:13.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:13.075+0000 7fe437aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:13.071 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:13.084+0000 7fb506769000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:13.071 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:13.085+0000 7fb506769000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:13.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:13.086+0000 7fe437aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:13.147 INFO:teuthology.orchestra.run.smithi059.stderr:adjusted tunables profile to default 2024-04-01T13:30:13.159 INFO:tasks.ceph:check_enable_crimson: True 2024-04-01T13:30:13.159 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-01T13:30:13.159 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-01T13:30:13.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:13.277+0000 7f7b0e954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:13.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:13.287+0000 7f7b0e954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:13.357 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:30:13.371+0000 7f79745f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-01T13:30:13.357 INFO:tasks.ceph.mon.a.smithi059.stderr:Please be aware that this feature is experimental, untested, 2024-04-01T13:30:13.358 INFO:tasks.ceph.mon.a.smithi059.stderr:unsupported, and may result in data corruption, data loss, 2024-04-01T13:30:13.358 INFO:tasks.ceph.mon.a.smithi059.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-01T13:30:13.358 INFO:tasks.ceph.mon.a.smithi059.stderr:feature with important data. 2024-04-01T13:30:13.358 INFO:tasks.ceph.mon.a.smithi059.stderr: 2024-04-01T13:30:13.527 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:13.540+0000 7fb506769000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-01T13:30:13.578 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:13.592+0000 7fb506769000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-01T13:30:14.100 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:14.114+0000 7fb506769000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-01T13:30:14.152 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:14.166+0000 7fb506769000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-01T13:30:14.164 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-01T13:30:14.164 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-01T13:30:14.164 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-01T13:30:14.195 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-01T13:30:14.195 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-01T13:30:14.195 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:14.208+0000 7fb506769000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-01T13:30:14.246 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:14.260+0000 7fb506769000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-01T13:30:14.265 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-01T13:30:14.265 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-01T13:30:14.289 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:14.303+0000 7fb506769000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-01T13:30:14.292 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-01T13:30:14.292 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-01T13:30:14.359 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 6d8f0099-ddef-4ab2-8bbf-8fb9a032805e 0 2024-04-01T13:30:14.409 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:14.423+0000 7fb506769000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-01T13:30:14.482 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:14.496+0000 7fb506769000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-01T13:30:14.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-01T13:30:14.516+0000 7f48ce35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:14.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-01T13:30:14.525+0000 7f48ce35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:14.529 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:14.543+0000 7fb506769000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-01T13:30:14.580 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:14.594+0000 7fb506769000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-01T13:30:14.588 INFO:teuthology.orchestra.run.smithi087.stdout:0 2024-04-01T13:30:14.600 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 6800d5d2-a66b-4635-9100-e06da3c54f44 1 2024-04-01T13:30:14.629 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:14.642+0000 7fb506769000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-01T13:30:14.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-01T13:30:14.745+0000 7fbeefcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:14.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-01T13:30:14.754+0000 7fbeefcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:14.818 INFO:teuthology.orchestra.run.smithi087.stdout:1 2024-04-01T13:30:14.829 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 1a3e92de-e569-4465-a3bb-a14ea6b704a6 2 2024-04-01T13:30:14.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-01T13:30:14.947+0000 7f43c6c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:14.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-01T13:30:14.956+0000 7f43c6c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:15.018 INFO:teuthology.orchestra.run.smithi087.stdout:2 2024-04-01T13:30:15.020 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:15.034+0000 7fb506769000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-01T13:30:15.029 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 1d6f4971-3c5e-4334-9ce5-858400225e97 3 2024-04-01T13:30:15.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-01T13:30:15.146+0000 7fcac50ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:15.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-01T13:30:15.154+0000 7fcac50ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:15.152 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:15.165+0000 7fb506769000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-01T13:30:15.219 INFO:teuthology.orchestra.run.smithi087.stdout:3 2024-04-01T13:30:15.231 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-01T13:30:15.265 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:15.279+0000 7fb506769000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-01T13:30:15.268 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-01T13:30:15.268 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-01T13:30:15.310 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:15.324+0000 7fb506769000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-01T13:30:15.310 INFO:tasks.ceph.osd.0:Started 2024-04-01T13:30:15.311 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-01T13:30:15.344 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-01T13:30:15.344 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-01T13:30:15.386 INFO:tasks.ceph.osd.1:Started 2024-04-01T13:30:15.387 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-01T13:30:15.418 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-01T13:30:15.418 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-01T13:30:15.422 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:15.435+0000 7fb506769000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-01T13:30:15.423 INFO:tasks.ceph.osd.2:Started 2024-04-01T13:30:15.423 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-04-01T13:30:15.454 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-01T13:30:15.454 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-01T13:30:15.498 INFO:tasks.ceph.osd.3:Started 2024-04-01T13:30:15.499 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T13:30:15.525 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:15.539+0000 7fb506769000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-01T13:30:15.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:15.629+0000 7f3b98d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:15.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:15.639+0000 7f3b98d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:15.696 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:15.709+0000 7fb506769000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-01T13:30:15.696 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:30:15.696 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":7,"fsid":"2a626e12-75d1-4fa9-b59b-fc64723c30fe","created":"2024-04-01T13:30:13.072838+0000","modified":"2024-04-01T13:30:15.232514+0000","last_up_change":"0.000000","last_in_change":"2024-04-01T13:30:15.232514+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"6d8f0099-ddef-4ab2-8bbf-8fb9a032805e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6800d5d2-a66b-4635-9100-e06da3c54f44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1a3e92de-e569-4465-a3bb-a14ea6b704a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1d6f4971-3c5e-4334-9ce5-858400225e97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T13:30:15.707 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-01T13:30:15.707 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-01T13:30:15.756 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:15.770+0000 7fb506769000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-01T13:30:15.999 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:16.013+0000 7fb506769000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-01T13:30:16.110 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-01T13:30:16.216 INFO:teuthology.misc.health.smithi059.stderr:2024-04-01T13:30:16.225+0000 7fb83f66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:16.221 INFO:teuthology.misc.health.smithi059.stderr:2024-04-01T13:30:16.234+0000 7fb83f66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:16.291 INFO:teuthology.misc.health.smithi059.stdout: 2024-04-01T13:30:16.291 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":7,"fsid":"2a626e12-75d1-4fa9-b59b-fc64723c30fe","created":"2024-04-01T13:30:13.072838+0000","modified":"2024-04-01T13:30:15.232514+0000","last_up_change":"0.000000","last_in_change":"2024-04-01T13:30:15.232514+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"6d8f0099-ddef-4ab2-8bbf-8fb9a032805e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6800d5d2-a66b-4635-9100-e06da3c54f44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1a3e92de-e569-4465-a3bb-a14ea6b704a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1d6f4971-3c5e-4334-9ce5-858400225e97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T13:30:16.301 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-04-01T13:30:16.882 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:16.896+0000 7fb506769000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-01T13:30:17.158 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:17.172+0000 7fb506769000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-01T13:30:17.211 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:17.225+0000 7fb506769000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-01T13:30:17.341 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:17.354+0000 7fb506769000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-01T13:30:18.353 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:30:18.367+0000 7fb4a599a640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-01T13:30:22.705 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-01T13:30:22.819 INFO:teuthology.misc.health.smithi059.stderr:2024-04-01T13:30:22.829+0000 7f92b2e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:22.824 INFO:teuthology.misc.health.smithi059.stderr:2024-04-01T13:30:22.838+0000 7f92b2e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:23.024 INFO:teuthology.misc.health.smithi059.stdout: 2024-04-01T13:30:23.024 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"2a626e12-75d1-4fa9-b59b-fc64723c30fe","created":"2024-04-01T13:30:13.072838+0000","modified":"2024-04-01T13:30:22.378617+0000","last_up_change":"2024-04-01T13:30:20.368289+0000","last_in_change":"2024-04-01T13:30:15.232514+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T13:30:21.384810+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6d8f0099-ddef-4ab2-8bbf-8fb9a032805e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":411905682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":411905682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":411905682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6807","nonce":411905682}]},"public_addr":"172.21.15.59:6802/411905682","cluster_addr":"172.21.15.59:6801/411905682","heartbeat_back_addr":"172.21.15.59:6810/411905682","heartbeat_front_addr":"172.21.15.59:6807/411905682","state":["exists","up"]},{"osd":1,"uuid":"6800d5d2-a66b-4635-9100-e06da3c54f44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2240657276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":2240657276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":2240657276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2240657276}]},"public_addr":"172.21.15.59:6804/2240657276","cluster_addr":"172.21.15.59:6805/2240657276","heartbeat_back_addr":"172.21.15.59:6809/2240657276","heartbeat_front_addr":"172.21.15.59:6808/2240657276","state":["exists","up"]},{"osd":2,"uuid":"1a3e92de-e569-4465-a3bb-a14ea6b704a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4027514127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":4027514127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":4027514127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":4027514127}]},"public_addr":"172.21.15.87:6804/4027514127","cluster_addr":"172.21.15.87:6805/4027514127","heartbeat_back_addr":"172.21.15.87:6810/4027514127","heartbeat_front_addr":"172.21.15.87:6811/4027514127","state":["exists","up"]},{"osd":3,"uuid":"1d6f4971-3c5e-4334-9ce5-858400225e97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":4052372498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4052372498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":4052372498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4052372498}]},"public_addr":"172.21.15.87:6809/4052372498","cluster_addr":"172.21.15.87:6808/4052372498","heartbeat_back_addr":"172.21.15.87:6807/4052372498","heartbeat_front_addr":"172.21.15.87:6806/4052372498","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T13:30:23.035 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-04-01T13:30:23.036 INFO:tasks.ceph:Creating RBD pool 2024-04-01T13:30:23.036 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-01T13:30:23.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:23.155+0000 7fb10022d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:23.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:23.166+0000 7fb10022d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:23.368 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'rbd' created 2024-04-01T13:30:23.384 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-01T13:30:23.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:23.530+0000 7fdbdfd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:23.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:23.545+0000 7fdbdfd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:24.376 INFO:teuthology.orchestra.run.smithi059.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-01T13:30:24.395 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-01T13:30:24.396 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-01T13:30:24.396 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-01T13:30:24.396 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-01T13:30:24.396 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-01T13:30:24.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:24.559+0000 7f5f6fa20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:24.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:24.568+0000 7f5f6fa20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:24.809 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:30:24.821 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":1670700141},{"type":"v1","addr":"172.21.15.87:6801","nonce":1670700141}]},"active_addr":"172.21.15.87:6801/1670700141","active_change":"2024-04-01T13:30:17.357526+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":3860106081}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":1632166763}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":2022434838}]}]} 2024-04-01T13:30:24.825 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-01T13:30:24.825 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-01T13:30:24.825 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T13:30:24.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:24.952+0000 7f329c206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:24.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:24.962+0000 7f329c206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:25.150 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:30:25.150 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":12,"fsid":"2a626e12-75d1-4fa9-b59b-fc64723c30fe","created":"2024-04-01T13:30:13.072838+0000","modified":"2024-04-01T13:30:24.387700+0000","last_up_change":"2024-04-01T13:30:20.368289+0000","last_in_change":"2024-04-01T13:30:15.232514+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T13:30:21.384810+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-01T13:30:23.367687+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6d8f0099-ddef-4ab2-8bbf-8fb9a032805e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":411905682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":411905682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":411905682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6807","nonce":411905682}]},"public_addr":"172.21.15.59:6802/411905682","cluster_addr":"172.21.15.59:6801/411905682","heartbeat_back_addr":"172.21.15.59:6810/411905682","heartbeat_front_addr":"172.21.15.59:6807/411905682","state":["exists","up"]},{"osd":1,"uuid":"6800d5d2-a66b-4635-9100-e06da3c54f44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2240657276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":2240657276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":2240657276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2240657276}]},"public_addr":"172.21.15.59:6804/2240657276","cluster_addr":"172.21.15.59:6805/2240657276","heartbeat_back_addr":"172.21.15.59:6809/2240657276","heartbeat_front_addr":"172.21.15.59:6808/2240657276","state":["exists","up"]},{"osd":2,"uuid":"1a3e92de-e569-4465-a3bb-a14ea6b704a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4027514127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":4027514127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":4027514127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":4027514127}]},"public_addr":"172.21.15.87:6804/4027514127","cluster_addr":"172.21.15.87:6805/4027514127","heartbeat_back_addr":"172.21.15.87:6810/4027514127","heartbeat_front_addr":"172.21.15.87:6811/4027514127","state":["exists","up"]},{"osd":3,"uuid":"1d6f4971-3c5e-4334-9ce5-858400225e97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":4052372498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4052372498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":4052372498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4052372498}]},"public_addr":"172.21.15.87:6809/4052372498","cluster_addr":"172.21.15.87:6808/4052372498","heartbeat_back_addr":"172.21.15.87:6807/4052372498","heartbeat_front_addr":"172.21.15.87:6806/4052372498","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T13:30:25.162 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-01T13:30:25.163 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T13:30:25.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:25.285+0000 7fb16ae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:25.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:25.293+0000 7fb16ae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:25.487 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:30:25.487 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":13,"fsid":"2a626e12-75d1-4fa9-b59b-fc64723c30fe","created":"2024-04-01T13:30:13.072838+0000","modified":"2024-04-01T13:30:25.390560+0000","last_up_change":"2024-04-01T13:30:20.368289+0000","last_in_change":"2024-04-01T13:30:15.232514+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T13:30:21.384810+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-01T13:30:23.367687+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6d8f0099-ddef-4ab2-8bbf-8fb9a032805e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":411905682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":411905682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":411905682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6807","nonce":411905682}]},"public_addr":"172.21.15.59:6802/411905682","cluster_addr":"172.21.15.59:6801/411905682","heartbeat_back_addr":"172.21.15.59:6810/411905682","heartbeat_front_addr":"172.21.15.59:6807/411905682","state":["exists","up"]},{"osd":1,"uuid":"6800d5d2-a66b-4635-9100-e06da3c54f44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2240657276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":2240657276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":2240657276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2240657276}]},"public_addr":"172.21.15.59:6804/2240657276","cluster_addr":"172.21.15.59:6805/2240657276","heartbeat_back_addr":"172.21.15.59:6809/2240657276","heartbeat_front_addr":"172.21.15.59:6808/2240657276","state":["exists","up"]},{"osd":2,"uuid":"1a3e92de-e569-4465-a3bb-a14ea6b704a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4027514127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":4027514127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":4027514127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":4027514127}]},"public_addr":"172.21.15.87:6804/4027514127","cluster_addr":"172.21.15.87:6805/4027514127","heartbeat_back_addr":"172.21.15.87:6810/4027514127","heartbeat_front_addr":"172.21.15.87:6811/4027514127","state":["exists","up"]},{"osd":3,"uuid":"1d6f4971-3c5e-4334-9ce5-858400225e97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":4052372498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4052372498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":4052372498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4052372498}]},"public_addr":"172.21.15.87:6809/4052372498","cluster_addr":"172.21.15.87:6808/4052372498","heartbeat_back_addr":"172.21.15.87:6807/4052372498","heartbeat_front_addr":"172.21.15.87:6806/4052372498","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T13:30:25.501 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-01T13:30:25.501 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-01T13:30:25.501 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-01T13:30:25.502 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-01T13:30:25.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:25.638+0000 7f7bc0b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:25.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:25.638+0000 7f8ede42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:25.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:25.638+0000 7f45be91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:25.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:25.638+0000 7f7a8e42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:25.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:25.648+0000 7f8ede42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:25.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:25.648+0000 7f45be91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:25.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:25.649+0000 7f7bc0b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:25.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:25.649+0000 7f7a8e42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:25.670 INFO:teuthology.orchestra.run.smithi059.stdout:38654705669 2024-04-01T13:30:25.670 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-01T13:30:25.673 INFO:teuthology.orchestra.run.smithi059.stdout:38654705669 2024-04-01T13:30:25.673 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-01T13:30:25.674 INFO:teuthology.orchestra.run.smithi059.stdout:38654705669 2024-04-01T13:30:25.675 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-01T13:30:25.678 INFO:teuthology.orchestra.run.smithi059.stdout:38654705669 2024-04-01T13:30:25.678 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-01T13:30:25.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:25.796+0000 7f6fbdcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:25.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:25.808+0000 7f6fbdcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:25.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:25.847+0000 7fa154c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:25.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:25.857+0000 7fa154c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:25.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:25.852+0000 7ff935b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:25.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:25.866+0000 7ff935b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:25.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:25.895+0000 7fd15d2ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:25.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:25.905+0000 7fd15d2ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:26.022 INFO:teuthology.orchestra.run.smithi059.stdout:38654705669 2024-04-01T13:30:26.035 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.3 2024-04-01T13:30:26.035 DEBUG:teuthology.parallel:result is None 2024-04-01T13:30:26.051 INFO:teuthology.orchestra.run.smithi059.stdout:38654705669 2024-04-01T13:30:26.064 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.0 2024-04-01T13:30:26.064 INFO:teuthology.orchestra.run.smithi059.stdout:38654705669 2024-04-01T13:30:26.064 DEBUG:teuthology.parallel:result is None 2024-04-01T13:30:26.077 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.2 2024-04-01T13:30:26.077 DEBUG:teuthology.parallel:result is None 2024-04-01T13:30:26.094 INFO:teuthology.orchestra.run.smithi059.stdout:38654705669 2024-04-01T13:30:26.106 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.1 2024-04-01T13:30:26.106 DEBUG:teuthology.parallel:result is None 2024-04-01T13:30:26.106 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-01T13:30:26.107 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T13:30:26.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:26.269+0000 7fb572cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:26.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:26.277+0000 7fb572cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:26.462 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:30:26.462 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-01T13:30:26.474 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-01T13:30:25.400304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107912,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379448,"statfs":{"total":386547056640,"available":386436554752,"internally_reserved":0,"allocated":593920,"data_stored":1870024,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.494295+0000","last_change":"2024-04-01T13:30:24.495962+0000","last_active":"2024-04-01T13:30:24.494295+0000","last_peered":"2024-04-01T13:30:24.494295+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.489422+0000","last_became_peered":"2024-04-01T13:30:24.489422+0000","last_unstale":"2024-04-01T13:30:24.494295+0000","last_undegraded":"2024-04-01T13:30:24.494295+0000","last_fullsized":"2024-04-01T13:30:24.494295+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.525265+0000","last_change":"2024-04-01T13:30:24.526346+0000","last_active":"2024-04-01T13:30:24.525265+0000","last_peered":"2024-04-01T13:30:24.525265+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.521930+0000","last_became_peered":"2024-04-01T13:30:24.521930+0000","last_unstale":"2024-04-01T13:30:24.525265+0000","last_undegraded":"2024-04-01T13:30:24.525265+0000","last_fullsized":"2024-04-01T13:30:24.525265+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.480277+0000","last_change":"2024-04-01T13:30:24.481500+0000","last_active":"2024-04-01T13:30:24.480277+0000","last_peered":"2024-04-01T13:30:24.480277+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.476810+0000","last_became_peered":"2024-04-01T13:30:24.476810+0000","last_unstale":"2024-04-01T13:30:24.480277+0000","last_undegraded":"2024-04-01T13:30:24.480277+0000","last_fullsized":"2024-04-01T13:30:24.480277+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.498967+0000","last_change":"2024-04-01T13:30:24.499851+0000","last_active":"2024-04-01T13:30:24.498967+0000","last_peered":"2024-04-01T13:30:24.498967+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.495368+0000","last_became_peered":"2024-04-01T13:30:24.495368+0000","last_unstale":"2024-04-01T13:30:24.498967+0000","last_undegraded":"2024-04-01T13:30:24.498967+0000","last_fullsized":"2024-04-01T13:30:24.498967+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.478572+0000","last_change":"2024-04-01T13:30:24.480593+0000","last_active":"2024-04-01T13:30:24.478572+0000","last_peered":"2024-04-01T13:30:24.478572+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.472751+0000","last_became_peered":"2024-04-01T13:30:24.472751+0000","last_unstale":"2024-04-01T13:30:24.478572+0000","last_undegraded":"2024-04-01T13:30:24.478572+0000","last_fullsized":"2024-04-01T13:30:24.478572+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.463161+0000","last_change":"2024-04-01T13:30:24.464855+0000","last_active":"2024-04-01T13:30:24.463161+0000","last_peered":"2024-04-01T13:30:24.463161+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.458263+0000","last_became_peered":"2024-04-01T13:30:24.458263+0000","last_unstale":"2024-04-01T13:30:24.463161+0000","last_undegraded":"2024-04-01T13:30:24.463161+0000","last_fullsized":"2024-04-01T13:30:24.463161+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.493074+0000","last_change":"2024-04-01T13:30:24.494919+0000","last_active":"2024-04-01T13:30:24.493074+0000","last_peered":"2024-04-01T13:30:24.493074+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.486637+0000","last_became_peered":"2024-04-01T13:30:24.486637+0000","last_unstale":"2024-04-01T13:30:24.493074+0000","last_undegraded":"2024-04-01T13:30:24.493074+0000","last_fullsized":"2024-04-01T13:30:24.493074+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.492008+0000","last_change":"2024-04-01T13:30:24.493976+0000","last_active":"2024-04-01T13:30:24.492008+0000","last_peered":"2024-04-01T13:30:24.492008+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.483528+0000","last_became_peered":"2024-04-01T13:30:24.483528+0000","last_unstale":"2024-04-01T13:30:24.492008+0000","last_undegraded":"2024-04-01T13:30:24.492008+0000","last_fullsized":"2024-04-01T13:30:24.492008+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.962629+0000","last_change":"2024-04-01T13:30:23.446271+0000","last_active":"2024-04-01T13:30:24.962629+0000","last_peered":"2024-04-01T13:30:24.962629+0000","last_clean":"2024-04-01T13:30:24.962629+0000","last_became_active":"2024-04-01T13:30:23.439285+0000","last_became_peered":"2024-04-01T13:30:23.439285+0000","last_unstale":"2024-04-01T13:30:24.962629+0000","last_undegraded":"2024-04-01T13:30:24.962629+0000","last_fullsized":"2024-04-01T13:30:24.962629+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_clean_scrub_stamp":"2024-04-01T13:30:22.378617+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-01T13:30:26.475 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T13:30:26.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:26.598+0000 7f5235424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:26.606+0000 7f5235424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:26.788 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:30:26.788 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-01T13:30:26.800 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-01T13:30:25.400304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107912,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379448,"statfs":{"total":386547056640,"available":386436554752,"internally_reserved":0,"allocated":593920,"data_stored":1870024,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.494295+0000","last_change":"2024-04-01T13:30:24.495962+0000","last_active":"2024-04-01T13:30:24.494295+0000","last_peered":"2024-04-01T13:30:24.494295+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.489422+0000","last_became_peered":"2024-04-01T13:30:24.489422+0000","last_unstale":"2024-04-01T13:30:24.494295+0000","last_undegraded":"2024-04-01T13:30:24.494295+0000","last_fullsized":"2024-04-01T13:30:24.494295+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.525265+0000","last_change":"2024-04-01T13:30:24.526346+0000","last_active":"2024-04-01T13:30:24.525265+0000","last_peered":"2024-04-01T13:30:24.525265+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.521930+0000","last_became_peered":"2024-04-01T13:30:24.521930+0000","last_unstale":"2024-04-01T13:30:24.525265+0000","last_undegraded":"2024-04-01T13:30:24.525265+0000","last_fullsized":"2024-04-01T13:30:24.525265+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.480277+0000","last_change":"2024-04-01T13:30:24.481500+0000","last_active":"2024-04-01T13:30:24.480277+0000","last_peered":"2024-04-01T13:30:24.480277+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.476810+0000","last_became_peered":"2024-04-01T13:30:24.476810+0000","last_unstale":"2024-04-01T13:30:24.480277+0000","last_undegraded":"2024-04-01T13:30:24.480277+0000","last_fullsized":"2024-04-01T13:30:24.480277+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.498967+0000","last_change":"2024-04-01T13:30:24.499851+0000","last_active":"2024-04-01T13:30:24.498967+0000","last_peered":"2024-04-01T13:30:24.498967+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.495368+0000","last_became_peered":"2024-04-01T13:30:24.495368+0000","last_unstale":"2024-04-01T13:30:24.498967+0000","last_undegraded":"2024-04-01T13:30:24.498967+0000","last_fullsized":"2024-04-01T13:30:24.498967+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.478572+0000","last_change":"2024-04-01T13:30:24.480593+0000","last_active":"2024-04-01T13:30:24.478572+0000","last_peered":"2024-04-01T13:30:24.478572+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.472751+0000","last_became_peered":"2024-04-01T13:30:24.472751+0000","last_unstale":"2024-04-01T13:30:24.478572+0000","last_undegraded":"2024-04-01T13:30:24.478572+0000","last_fullsized":"2024-04-01T13:30:24.478572+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.463161+0000","last_change":"2024-04-01T13:30:24.464855+0000","last_active":"2024-04-01T13:30:24.463161+0000","last_peered":"2024-04-01T13:30:24.463161+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.458263+0000","last_became_peered":"2024-04-01T13:30:24.458263+0000","last_unstale":"2024-04-01T13:30:24.463161+0000","last_undegraded":"2024-04-01T13:30:24.463161+0000","last_fullsized":"2024-04-01T13:30:24.463161+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.493074+0000","last_change":"2024-04-01T13:30:24.494919+0000","last_active":"2024-04-01T13:30:24.493074+0000","last_peered":"2024-04-01T13:30:24.493074+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.486637+0000","last_became_peered":"2024-04-01T13:30:24.486637+0000","last_unstale":"2024-04-01T13:30:24.493074+0000","last_undegraded":"2024-04-01T13:30:24.493074+0000","last_fullsized":"2024-04-01T13:30:24.493074+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.492008+0000","last_change":"2024-04-01T13:30:24.493976+0000","last_active":"2024-04-01T13:30:24.492008+0000","last_peered":"2024-04-01T13:30:24.492008+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.483528+0000","last_became_peered":"2024-04-01T13:30:24.483528+0000","last_unstale":"2024-04-01T13:30:24.492008+0000","last_undegraded":"2024-04-01T13:30:24.492008+0000","last_fullsized":"2024-04-01T13:30:24.492008+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.962629+0000","last_change":"2024-04-01T13:30:23.446271+0000","last_active":"2024-04-01T13:30:24.962629+0000","last_peered":"2024-04-01T13:30:24.962629+0000","last_clean":"2024-04-01T13:30:24.962629+0000","last_became_active":"2024-04-01T13:30:23.439285+0000","last_became_peered":"2024-04-01T13:30:23.439285+0000","last_unstale":"2024-04-01T13:30:24.962629+0000","last_undegraded":"2024-04-01T13:30:24.962629+0000","last_fullsized":"2024-04-01T13:30:24.962629+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_clean_scrub_stamp":"2024-04-01T13:30:22.378617+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-01T13:30:26.801 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-01T13:30:26.801 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-01T13:30:26.801 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-01T13:30:26.801 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-01T13:30:26.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:26.922+0000 7f994c024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:26.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:26.930+0000 7f994c024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:27.155 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:30:27.155 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-01T13:30:27.168 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-01T13:30:27.168 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-01T13:30:27.178 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-01T13:30:27.178 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-matanb-crimson-testing-thrash-options'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'afa73a9e21b8fb0610f8af7266f1975be37c0a69'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'conf': {'osd': {'osd deep scrub update digest min age': 0}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'afa73a9e21b8fb0610f8af7266f1975be37c0a69'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-matanb-crimson-testing-thrash-options', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'afa73a9e21b8fb0610f8af7266f1975be37c0a69'}} 2024-04-01T13:30:27.178 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-01T13:30:27.179 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-01T13:30:27.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:27.300+0000 7fe4b0417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:27.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:27.312+0000 7fe4b0417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:27.518 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-01T13:30:27.518 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-01T13:30:27.518 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T13:30:27.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:27.639+0000 7fda798ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:27.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:27.647+0000 7fda798ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:27.828 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:30:27.828 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-01T13:30:27.840 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-01T13:30:27.365356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107912,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379448,"statfs":{"total":386547056640,"available":386436554752,"internally_reserved":0,"allocated":593920,"data_stored":1870024,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.965052"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T13:30:25.407393+0000","last_change":"2024-04-01T13:30:24.495962+0000","last_active":"2024-04-01T13:30:25.407393+0000","last_peered":"2024-04-01T13:30:25.407393+0000","last_clean":"2024-04-01T13:30:25.407393+0000","last_became_active":"2024-04-01T13:30:24.489422+0000","last_became_peered":"2024-04-01T13:30:24.489422+0000","last_unstale":"2024-04-01T13:30:25.407393+0000","last_undegraded":"2024-04-01T13:30:25.407393+0000","last_fullsized":"2024-04-01T13:30:25.407393+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.525265+0000","last_change":"2024-04-01T13:30:24.526346+0000","last_active":"2024-04-01T13:30:24.525265+0000","last_peered":"2024-04-01T13:30:24.525265+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.521930+0000","last_became_peered":"2024-04-01T13:30:24.521930+0000","last_unstale":"2024-04-01T13:30:24.525265+0000","last_undegraded":"2024-04-01T13:30:24.525265+0000","last_fullsized":"2024-04-01T13:30:24.525265+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.480277+0000","last_change":"2024-04-01T13:30:24.481500+0000","last_active":"2024-04-01T13:30:24.480277+0000","last_peered":"2024-04-01T13:30:24.480277+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.476810+0000","last_became_peered":"2024-04-01T13:30:24.476810+0000","last_unstale":"2024-04-01T13:30:24.480277+0000","last_undegraded":"2024-04-01T13:30:24.480277+0000","last_fullsized":"2024-04-01T13:30:24.480277+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.498967+0000","last_change":"2024-04-01T13:30:24.499851+0000","last_active":"2024-04-01T13:30:24.498967+0000","last_peered":"2024-04-01T13:30:24.498967+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.495368+0000","last_became_peered":"2024-04-01T13:30:24.495368+0000","last_unstale":"2024-04-01T13:30:24.498967+0000","last_undegraded":"2024-04-01T13:30:24.498967+0000","last_fullsized":"2024-04-01T13:30:24.498967+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.478572+0000","last_change":"2024-04-01T13:30:24.480593+0000","last_active":"2024-04-01T13:30:24.478572+0000","last_peered":"2024-04-01T13:30:24.478572+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.472751+0000","last_became_peered":"2024-04-01T13:30:24.472751+0000","last_unstale":"2024-04-01T13:30:24.478572+0000","last_undegraded":"2024-04-01T13:30:24.478572+0000","last_fullsized":"2024-04-01T13:30:24.478572+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T13:30:25.404306+0000","last_change":"2024-04-01T13:30:24.464855+0000","last_active":"2024-04-01T13:30:25.404306+0000","last_peered":"2024-04-01T13:30:25.404306+0000","last_clean":"2024-04-01T13:30:25.404306+0000","last_became_active":"2024-04-01T13:30:24.458263+0000","last_became_peered":"2024-04-01T13:30:24.458263+0000","last_unstale":"2024-04-01T13:30:25.404306+0000","last_undegraded":"2024-04-01T13:30:25.404306+0000","last_fullsized":"2024-04-01T13:30:25.404306+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.493074+0000","last_change":"2024-04-01T13:30:24.494919+0000","last_active":"2024-04-01T13:30:24.493074+0000","last_peered":"2024-04-01T13:30:24.493074+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.486637+0000","last_became_peered":"2024-04-01T13:30:24.486637+0000","last_unstale":"2024-04-01T13:30:24.493074+0000","last_undegraded":"2024-04-01T13:30:24.493074+0000","last_fullsized":"2024-04-01T13:30:24.493074+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T13:30:25.408144+0000","last_change":"2024-04-01T13:30:24.493976+0000","last_active":"2024-04-01T13:30:25.408144+0000","last_peered":"2024-04-01T13:30:25.408144+0000","last_clean":"2024-04-01T13:30:25.408144+0000","last_became_active":"2024-04-01T13:30:24.483528+0000","last_became_peered":"2024-04-01T13:30:24.483528+0000","last_unstale":"2024-04-01T13:30:25.408144+0000","last_undegraded":"2024-04-01T13:30:25.408144+0000","last_fullsized":"2024-04-01T13:30:25.408144+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.962629+0000","last_change":"2024-04-01T13:30:23.446271+0000","last_active":"2024-04-01T13:30:24.962629+0000","last_peered":"2024-04-01T13:30:24.962629+0000","last_clean":"2024-04-01T13:30:24.962629+0000","last_became_active":"2024-04-01T13:30:23.439285+0000","last_became_peered":"2024-04-01T13:30:23.439285+0000","last_unstale":"2024-04-01T13:30:24.962629+0000","last_undegraded":"2024-04-01T13:30:24.962629+0000","last_fullsized":"2024-04-01T13:30:24.962629+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_clean_scrub_stamp":"2024-04-01T13:30:22.378617+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-01T13:30:27.841 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T13:30:27.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:27.961+0000 7ff278c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:27.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:27.970+0000 7ff278c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:28.151 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:30:28.151 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-01T13:30:28.163 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-01T13:30:27.365356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107912,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379448,"statfs":{"total":386547056640,"available":386436554752,"internally_reserved":0,"allocated":593920,"data_stored":1870024,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.965052"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T13:30:25.407393+0000","last_change":"2024-04-01T13:30:24.495962+0000","last_active":"2024-04-01T13:30:25.407393+0000","last_peered":"2024-04-01T13:30:25.407393+0000","last_clean":"2024-04-01T13:30:25.407393+0000","last_became_active":"2024-04-01T13:30:24.489422+0000","last_became_peered":"2024-04-01T13:30:24.489422+0000","last_unstale":"2024-04-01T13:30:25.407393+0000","last_undegraded":"2024-04-01T13:30:25.407393+0000","last_fullsized":"2024-04-01T13:30:25.407393+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.525265+0000","last_change":"2024-04-01T13:30:24.526346+0000","last_active":"2024-04-01T13:30:24.525265+0000","last_peered":"2024-04-01T13:30:24.525265+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.521930+0000","last_became_peered":"2024-04-01T13:30:24.521930+0000","last_unstale":"2024-04-01T13:30:24.525265+0000","last_undegraded":"2024-04-01T13:30:24.525265+0000","last_fullsized":"2024-04-01T13:30:24.525265+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.480277+0000","last_change":"2024-04-01T13:30:24.481500+0000","last_active":"2024-04-01T13:30:24.480277+0000","last_peered":"2024-04-01T13:30:24.480277+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.476810+0000","last_became_peered":"2024-04-01T13:30:24.476810+0000","last_unstale":"2024-04-01T13:30:24.480277+0000","last_undegraded":"2024-04-01T13:30:24.480277+0000","last_fullsized":"2024-04-01T13:30:24.480277+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.498967+0000","last_change":"2024-04-01T13:30:24.499851+0000","last_active":"2024-04-01T13:30:24.498967+0000","last_peered":"2024-04-01T13:30:24.498967+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.495368+0000","last_became_peered":"2024-04-01T13:30:24.495368+0000","last_unstale":"2024-04-01T13:30:24.498967+0000","last_undegraded":"2024-04-01T13:30:24.498967+0000","last_fullsized":"2024-04-01T13:30:24.498967+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.478572+0000","last_change":"2024-04-01T13:30:24.480593+0000","last_active":"2024-04-01T13:30:24.478572+0000","last_peered":"2024-04-01T13:30:24.478572+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.472751+0000","last_became_peered":"2024-04-01T13:30:24.472751+0000","last_unstale":"2024-04-01T13:30:24.478572+0000","last_undegraded":"2024-04-01T13:30:24.478572+0000","last_fullsized":"2024-04-01T13:30:24.478572+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T13:30:25.404306+0000","last_change":"2024-04-01T13:30:24.464855+0000","last_active":"2024-04-01T13:30:25.404306+0000","last_peered":"2024-04-01T13:30:25.404306+0000","last_clean":"2024-04-01T13:30:25.404306+0000","last_became_active":"2024-04-01T13:30:24.458263+0000","last_became_peered":"2024-04-01T13:30:24.458263+0000","last_unstale":"2024-04-01T13:30:25.404306+0000","last_undegraded":"2024-04-01T13:30:25.404306+0000","last_fullsized":"2024-04-01T13:30:25.404306+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.493074+0000","last_change":"2024-04-01T13:30:24.494919+0000","last_active":"2024-04-01T13:30:24.493074+0000","last_peered":"2024-04-01T13:30:24.493074+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.486637+0000","last_became_peered":"2024-04-01T13:30:24.486637+0000","last_unstale":"2024-04-01T13:30:24.493074+0000","last_undegraded":"2024-04-01T13:30:24.493074+0000","last_fullsized":"2024-04-01T13:30:24.493074+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T13:30:25.408144+0000","last_change":"2024-04-01T13:30:24.493976+0000","last_active":"2024-04-01T13:30:25.408144+0000","last_peered":"2024-04-01T13:30:25.408144+0000","last_clean":"2024-04-01T13:30:25.408144+0000","last_became_active":"2024-04-01T13:30:24.483528+0000","last_became_peered":"2024-04-01T13:30:24.483528+0000","last_unstale":"2024-04-01T13:30:25.408144+0000","last_undegraded":"2024-04-01T13:30:25.408144+0000","last_fullsized":"2024-04-01T13:30:25.408144+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.962629+0000","last_change":"2024-04-01T13:30:23.446271+0000","last_active":"2024-04-01T13:30:24.962629+0000","last_peered":"2024-04-01T13:30:24.962629+0000","last_clean":"2024-04-01T13:30:24.962629+0000","last_became_active":"2024-04-01T13:30:23.439285+0000","last_became_peered":"2024-04-01T13:30:23.439285+0000","last_unstale":"2024-04-01T13:30:24.962629+0000","last_undegraded":"2024-04-01T13:30:24.962629+0000","last_fullsized":"2024-04-01T13:30:24.962629+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_clean_scrub_stamp":"2024-04-01T13:30:22.378617+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-01T13:30:28.164 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-01T13:30:28.164 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-01T13:30:28.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:28.286+0000 7faf1fd54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:28.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:28.294+0000 7faf1fd54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:28.481 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 13 2024-04-01T13:30:28.481 INFO:teuthology.orchestra.run.smithi059.stdout:fsid 2a626e12-75d1-4fa9-b59b-fc64723c30fe 2024-04-01T13:30:28.481 INFO:teuthology.orchestra.run.smithi059.stdout:created 2024-04-01T13:30:13.072838+0000 2024-04-01T13:30:28.482 INFO:teuthology.orchestra.run.smithi059.stdout:modified 2024-04-01T13:30:25.390560+0000 2024-04-01T13:30:28.482 INFO:teuthology.orchestra.run.smithi059.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-01T13:30:28.482 INFO:teuthology.orchestra.run.smithi059.stdout:crush_version 3 2024-04-01T13:30:28.482 INFO:teuthology.orchestra.run.smithi059.stdout:full_ratio 0.95 2024-04-01T13:30:28.482 INFO:teuthology.orchestra.run.smithi059.stdout:backfillfull_ratio 0.9 2024-04-01T13:30:28.482 INFO:teuthology.orchestra.run.smithi059.stdout:nearfull_ratio 0.85 2024-04-01T13:30:28.482 INFO:teuthology.orchestra.run.smithi059.stdout:require_min_compat_client luminous 2024-04-01T13:30:28.482 INFO:teuthology.orchestra.run.smithi059.stdout:min_compat_client jewel 2024-04-01T13:30:28.482 INFO:teuthology.orchestra.run.smithi059.stdout:require_osd_release squid 2024-04-01T13:30:28.482 INFO:teuthology.orchestra.run.smithi059.stdout:stretch_mode_enabled false 2024-04-01T13:30:28.482 INFO:teuthology.orchestra.run.smithi059.stdout:allow_crimson=true 2024-04-01T13:30:28.482 INFO:teuthology.orchestra.run.smithi059.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 12 flags hashpspool,nopgchange,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2024-04-01T13:30:28.482 INFO:teuthology.orchestra.run.smithi059.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 13 flags hashpspool,nopgchange,crimson stripe_width 0 application rbd read_balance_score 1.50 2024-04-01T13:30:28.482 INFO:teuthology.orchestra.run.smithi059.stdout:max_osd 4 2024-04-01T13:30:28.482 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.59:6802/411905682 v2:172.21.15.59:6801/411905682 exists,up 6d8f0099-ddef-4ab2-8bbf-8fb9a032805e 2024-04-01T13:30:28.483 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.59:6804/2240657276 v2:172.21.15.59:6805/2240657276 exists,up 6800d5d2-a66b-4635-9100-e06da3c54f44 2024-04-01T13:30:28.483 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6804/4027514127 v2:172.21.15.87:6805/4027514127 exists,up 1a3e92de-e569-4465-a3bb-a14ea6b704a6 2024-04-01T13:30:28.483 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6809/4052372498 v2:172.21.15.87:6808/4052372498 exists,up 1d6f4971-3c5e-4334-9ce5-858400225e97 2024-04-01T13:30:28.494 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.59:6802/411905682 v2:172.21.15.59:6801/411905682 exists,up 6d8f0099-ddef-4ab2-8bbf-8fb9a032805e', 'osd.1 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.59:6804/2240657276 v2:172.21.15.59:6805/2240657276 exists,up 6800d5d2-a66b-4635-9100-e06da3c54f44', 'osd.2 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6804/4027514127 v2:172.21.15.87:6805/4027514127 exists,up 1a3e92de-e569-4465-a3bb-a14ea6b704a6', 'osd.3 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6809/4052372498 v2:172.21.15.87:6808/4052372498 exists,up 1d6f4971-3c5e-4334-9ce5-858400225e97'] 2024-04-01T13:30:28.494 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-01T13:30:28.603 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-01T13:30:28.603 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "mon.a", 2024-04-01T13:30:28.603 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster": "ceph", 2024-04-01T13:30:28.603 INFO:teuthology.orchestra.run.smithi059.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-01T13:30:28.603 INFO:teuthology.orchestra.run.smithi059.stdout: "admin_socket_mode": "", 2024-04-01T13:30:28.603 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-01T13:30:28.603 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_client_required": "cephx, none", 2024-04-01T13:30:28.603 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_cluster_required": "cephx", 2024-04-01T13:30:28.603 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_debug": "true", 2024-04-01T13:30:28.603 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-01T13:30:28.604 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-01T13:30:28.604 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_service_required": "cephx", 2024-04-01T13:30:28.604 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-01T13:30:28.604 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_supported": "cephx", 2024-04-01T13:30:28.604 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_aio": "true", 2024-04-01T13:30:28.604 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-01T13:30:28.604 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_aio_poll_ms": "250", 2024-04-01T13:30:28.604 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_aio_reap_max": "16", 2024-04-01T13:30:28.604 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_async_discard_threads": "0", 2024-04-01T13:30:28.604 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_block_size": "4096", 2024-04-01T13:30:28.604 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_debug_aio": "false", 2024-04-01T13:30:28.604 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-01T13:30:28.604 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-01T13:30:28.604 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_debug_inflight_ios": "false", 2024-04-01T13:30:28.604 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_enable_discard": "false", 2024-04-01T13:30:28.605 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_flock_retry": "3", 2024-04-01T13:30:28.605 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-01T13:30:28.605 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_inject_crash": "0", 2024-04-01T13:30:28.605 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-01T13:30:28.605 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_ioring": "false", 2024-04-01T13:30:28.605 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_ioring_hipri": "false", 2024-04-01T13:30:28.605 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-01T13:30:28.605 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-01T13:30:28.605 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-01T13:30:28.605 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-01T13:30:28.605 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_type": "", 2024-04-01T13:30:28.605 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_alloc_size": "1048576", 2024-04-01T13:30:28.605 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_allocator": "hybrid", 2024-04-01T13:30:28.605 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_buffered_io": "true", 2024-04-01T13:30:28.605 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_check_for_zeros": "false", 2024-04-01T13:30:28.606 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-01T13:30:28.606 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-01T13:30:28.606 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_compact_log_sync": "false", 2024-04-01T13:30:28.606 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-01T13:30:28.606 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-01T13:30:28.606 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-01T13:30:28.606 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-01T13:30:28.606 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_max_log_runway": "4194304", 2024-04-01T13:30:28.606 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_max_prefetch": "1048576", 2024-04-01T13:30:28.606 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_min_flush_size": "524288", 2024-04-01T13:30:28.606 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_min_log_runway": "1048576", 2024-04-01T13:30:28.606 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_replay_recovery": "false", 2024-04-01T13:30:28.606 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-01T13:30:28.606 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-01T13:30:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_sync_write": "false", 2024-04-01T13:30:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-01T13:30:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-01T13:30:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-01T13:30:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_allocation_from_file": "true", 2024-04-01T13:30:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_allocator": "hybrid", 2024-04-01T13:30:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-01T13:30:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-01T13:30:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-01T13:30:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-01T13:30:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-01T13:30:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-01T13:30:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-01T13:30:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_create": "true", 2024-04-01T13:30:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_db_create": "false", 2024-04-01T13:30:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_db_path": "", 2024-04-01T13:30:28.608 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_db_size": "0", 2024-04-01T13:30:28.608 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_path": "", 2024-04-01T13:30:28.608 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_preallocate_file": "false", 2024-04-01T13:30:28.608 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_size": "107374182400", 2024-04-01T13:30:28.608 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_wal_create": "false", 2024-04-01T13:30:28.608 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_wal_path": "", 2024-04-01T13:30:28.608 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_wal_size": "100663296", 2024-04-01T13:30:28.608 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bluefs": "true", 2024-04-01T13:30:28.608 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-01T13:30:28.608 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-01T13:30:28.608 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-01T13:30:28.608 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-01T13:30:28.608 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-01T13:30:28.608 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-01T13:30:28.609 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-01T13:30:28.609 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-01T13:30:28.609 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_autotune": "true", 2024-04-01T13:30:28.609 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-01T13:30:28.609 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-01T13:30:28.609 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-01T13:30:28.609 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-01T13:30:28.609 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_size": "0", 2024-04-01T13:30:28.609 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-01T13:30:28.609 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-01T13:30:28.609 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-01T13:30:28.609 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-01T13:30:28.609 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_type": "2q", 2024-04-01T13:30:28.609 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-01T13:30:28.609 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_clone_cow": "true", 2024-04-01T13:30:28.610 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-01T13:30:28.610 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-01T13:30:28.610 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-01T13:30:28.610 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-01T13:30:28.610 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-01T13:30:28.610 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-01T13:30:28.610 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-01T13:30:28.610 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_mode": "none", 2024-04-01T13:30:28.610 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-01T13:30:28.610 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_csum_type": "crc32c", 2024-04-01T13:30:28.610 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-04-01T13:30:28.610 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-01T13:30:28.610 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_freelist": "false", 2024-04-01T13:30:28.610 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-01T13:30:28.611 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-01T13:30:28.611 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-01T13:30:28.611 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-01T13:30:28.611 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-01T13:30:28.611 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-01T13:30:28.611 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-01T13:30:28.611 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-01T13:30:28.611 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-01T13:30:28.611 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-01T13:30:28.611 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-01T13:30:28.611 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-01T13:30:28.611 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-01T13:30:28.611 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_small_allocations": "0", 2024-04-01T13:30:28.611 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-01T13:30:28.611 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_default_buffered_read": "true", 2024-04-01T13:30:28.612 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_default_buffered_write": "false", 2024-04-01T13:30:28.612 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-01T13:30:28.612 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-01T13:30:28.612 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-01T13:30:28.612 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-01T13:30:28.612 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-01T13:30:28.612 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-01T13:30:28.612 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-01T13:30:28.612 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-01T13:30:28.612 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-01T13:30:28.612 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fail_eio": "false", 2024-04-01T13:30:28.612 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-01T13:30:28.612 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-01T13:30:28.612 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-01T13:30:28.613 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-01T13:30:28.613 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-01T13:30:28.613 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-01T13:30:28.613 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_on_mount": "false", 2024-04-01T13:30:28.613 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-01T13:30:28.613 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_on_umount": "false", 2024-04-01T13:30:28.613 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-01T13:30:28.613 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-01T13:30:28.613 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-01T13:30:28.613 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-01T13:30:28.613 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-01T13:30:28.613 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-01T13:30:28.613 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-01T13:30:28.613 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-01T13:30:28.613 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_ignore_data_csum": "false", 2024-04-01T13:30:28.614 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-01T13:30:28.614 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-01T13:30:28.614 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-01T13:30:28.614 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-01T13:30:28.614 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_log_op_age": "5.000000", 2024-04-01T13:30:28.614 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_max_alloc_size": "0", 2024-04-01T13:30:28.614 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_max_blob_size": "0", 2024-04-01T13:30:28.614 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-01T13:30:28.614 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-01T13:30:28.614 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-01T13:30:28.614 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_max_deferred_txc": "32", 2024-04-01T13:30:28.614 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_min_alloc_size": "0", 2024-04-01T13:30:28.614 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-01T13:30:28.614 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-01T13:30:28.614 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_nid_prealloc": "1024", 2024-04-01T13:30:28.615 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-01T13:30:28.615 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-01T13:30:28.615 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-01T13:30:28.615 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-01T13:30:28.615 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_retry_disk_reads": "3", 2024-04-01T13:30:28.615 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_rocksdb_cf": "true", 2024-04-01T13:30:28.615 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-01T13:30:28.615 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-01T13:30:28.615 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-01T13:30:28.615 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-01T13:30:28.615 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-01T13:30:28.615 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-01T13:30:28.615 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_spdk_mem": "512", 2024-04-01T13:30:28.615 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-01T13:30:28.616 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-01T13:30:28.616 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-01T13:30:28.616 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-01T13:30:28.616 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-01T13:30:28.616 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-01T13:30:28.616 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-01T13:30:28.616 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_tracing": "false", 2024-04-01T13:30:28.616 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-01T13:30:28.616 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-01T13:30:28.616 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-01T13:30:28.616 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-01T13:30:28.616 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-01T13:30:28.616 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-01T13:30:28.616 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-01T13:30:28.616 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-01T13:30:28.616 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-01T13:30:28.617 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_zero_block_detection": "false", 2024-04-01T13:30:28.617 INFO:teuthology.orchestra.run.smithi059.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-01T13:30:28.617 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-01T13:30:28.617 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-01T13:30:28.617 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-01T13:30:28.617 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-01T13:30:28.617 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-01T13:30:28.617 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-01T13:30:28.617 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-01T13:30:28.617 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-01T13:30:28.617 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-01T13:30:28.617 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-01T13:30:28.617 INFO:teuthology.orchestra.run.smithi059.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-01T13:30:28.617 INFO:teuthology.orchestra.run.smithi059.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-01T13:30:28.617 INFO:teuthology.orchestra.run.smithi059.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-01T13:30:28.618 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_cluster_require_signatures": "false", 2024-04-01T13:30:28.618 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_cluster_require_version": "2", 2024-04-01T13:30:28.618 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_require_signatures": "false", 2024-04-01T13:30:28.618 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_require_version": "2", 2024-04-01T13:30:28.618 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_service_require_signatures": "false", 2024-04-01T13:30:28.618 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_service_require_version": "2", 2024-04-01T13:30:28.618 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_sign_messages": "true", 2024-04-01T13:30:28.618 INFO:teuthology.orchestra.run.smithi059.stdout: "chdir": "", 2024-04-01T13:30:28.618 INFO:teuthology.orchestra.run.smithi059.stdout: "client_acl_type": "", 2024-04-01T13:30:28.618 INFO:teuthology.orchestra.run.smithi059.stdout: "client_asio_thread_count": "2", 2024-04-01T13:30:28.618 INFO:teuthology.orchestra.run.smithi059.stdout: "client_cache_mid": "0.750000", 2024-04-01T13:30:28.618 INFO:teuthology.orchestra.run.smithi059.stdout: "client_cache_size": "16384", 2024-04-01T13:30:28.618 INFO:teuthology.orchestra.run.smithi059.stdout: "client_caps_release_delay": "5", 2024-04-01T13:30:28.618 INFO:teuthology.orchestra.run.smithi059.stdout: "client_check_pool_perm": "true", 2024-04-01T13:30:28.619 INFO:teuthology.orchestra.run.smithi059.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-01T13:30:28.619 INFO:teuthology.orchestra.run.smithi059.stdout: "client_debug_force_sync_read": "false", 2024-04-01T13:30:28.619 INFO:teuthology.orchestra.run.smithi059.stdout: "client_debug_getattr_caps": "false", 2024-04-01T13:30:28.619 INFO:teuthology.orchestra.run.smithi059.stdout: "client_debug_inject_tick_delay": "0", 2024-04-01T13:30:28.619 INFO:teuthology.orchestra.run.smithi059.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-01T13:30:28.619 INFO:teuthology.orchestra.run.smithi059.stdout: "client_die_on_failed_remount": "false", 2024-04-01T13:30:28.619 INFO:teuthology.orchestra.run.smithi059.stdout: "client_dirsize_rbytes": "true", 2024-04-01T13:30:28.619 INFO:teuthology.orchestra.run.smithi059.stdout: "client_force_lazyio": "false", 2024-04-01T13:30:28.619 INFO:teuthology.orchestra.run.smithi059.stdout: "client_fs": "", 2024-04-01T13:30:28.619 INFO:teuthology.orchestra.run.smithi059.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-01T13:30:28.619 INFO:teuthology.orchestra.run.smithi059.stdout: "client_inject_release_failure": "false", 2024-04-01T13:30:28.619 INFO:teuthology.orchestra.run.smithi059.stdout: "client_max_inline_size": "4096", 2024-04-01T13:30:28.619 INFO:teuthology.orchestra.run.smithi059.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-01T13:30:28.619 INFO:teuthology.orchestra.run.smithi059.stdout: "client_mds_namespace": "", 2024-04-01T13:30:28.619 INFO:teuthology.orchestra.run.smithi059.stdout: "client_metadata": "", 2024-04-01T13:30:28.620 INFO:teuthology.orchestra.run.smithi059.stdout: "client_mount_gid": "-1", 2024-04-01T13:30:28.620 INFO:teuthology.orchestra.run.smithi059.stdout: "client_mount_timeout": "300", 2024-04-01T13:30:28.620 INFO:teuthology.orchestra.run.smithi059.stdout: "client_mount_uid": "-1", 2024-04-01T13:30:28.620 INFO:teuthology.orchestra.run.smithi059.stdout: "client_mountpoint": "/", 2024-04-01T13:30:28.620 INFO:teuthology.orchestra.run.smithi059.stdout: "client_notify_timeout": "10", 2024-04-01T13:30:28.620 INFO:teuthology.orchestra.run.smithi059.stdout: "client_oc": "true", 2024-04-01T13:30:28.620 INFO:teuthology.orchestra.run.smithi059.stdout: "client_oc_max_dirty": "104857600", 2024-04-01T13:30:28.620 INFO:teuthology.orchestra.run.smithi059.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-01T13:30:28.620 INFO:teuthology.orchestra.run.smithi059.stdout: "client_oc_max_objects": "1000", 2024-04-01T13:30:28.620 INFO:teuthology.orchestra.run.smithi059.stdout: "client_oc_size": "209715200", 2024-04-01T13:30:28.620 INFO:teuthology.orchestra.run.smithi059.stdout: "client_oc_target_dirty": "8388608", 2024-04-01T13:30:28.620 INFO:teuthology.orchestra.run.smithi059.stdout: "client_permissions": "true", 2024-04-01T13:30:28.620 INFO:teuthology.orchestra.run.smithi059.stdout: "client_quota": "true", 2024-04-01T13:30:28.620 INFO:teuthology.orchestra.run.smithi059.stdout: "client_quota_df": "true", 2024-04-01T13:30:28.620 INFO:teuthology.orchestra.run.smithi059.stdout: "client_readahead_max_bytes": "0", 2024-04-01T13:30:28.620 INFO:teuthology.orchestra.run.smithi059.stdout: "client_readahead_max_periods": "4", 2024-04-01T13:30:28.621 INFO:teuthology.orchestra.run.smithi059.stdout: "client_readahead_min": "131072", 2024-04-01T13:30:28.621 INFO:teuthology.orchestra.run.smithi059.stdout: "client_reconnect_stale": "false", 2024-04-01T13:30:28.621 INFO:teuthology.orchestra.run.smithi059.stdout: "client_shutdown_timeout": "30", 2024-04-01T13:30:28.621 INFO:teuthology.orchestra.run.smithi059.stdout: "client_snapdir": ".snap", 2024-04-01T13:30:28.621 INFO:teuthology.orchestra.run.smithi059.stdout: "client_tick_interval": "1", 2024-04-01T13:30:28.621 INFO:teuthology.orchestra.run.smithi059.stdout: "client_trace": "", 2024-04-01T13:30:28.621 INFO:teuthology.orchestra.run.smithi059.stdout: "client_try_dentry_invalidate": "false", 2024-04-01T13:30:28.621 INFO:teuthology.orchestra.run.smithi059.stdout: "client_use_faked_inos": "false", 2024-04-01T13:30:28.621 INFO:teuthology.orchestra.run.smithi059.stdout: "client_use_random_mds": "false", 2024-04-01T13:30:28.621 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_graylog": "false", 2024-04-01T13:30:28.621 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-01T13:30:28.621 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_graylog_port": "12201", 2024-04-01T13:30:28.621 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_monitors": "default=true", 2024-04-01T13:30:28.621 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_syslog": "false", 2024-04-01T13:30:28.621 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-01T13:30:28.622 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_syslog_level": "info", 2024-04-01T13:30:28.622 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "-", 2024-04-01T13:30:28.622 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_network": "", 2024-04-01T13:30:28.622 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_network_interface": "", 2024-04-01T13:30:28.622 INFO:teuthology.orchestra.run.smithi059.stdout: "compressor_zlib_isal": "false", 2024-04-01T13:30:28.622 INFO:teuthology.orchestra.run.smithi059.stdout: "compressor_zlib_level": "5", 2024-04-01T13:30:28.622 INFO:teuthology.orchestra.run.smithi059.stdout: "compressor_zlib_winsize": "-15", 2024-04-01T13:30:28.622 INFO:teuthology.orchestra.run.smithi059.stdout: "compressor_zstd_level": "1", 2024-04-01T13:30:28.622 INFO:teuthology.orchestra.run.smithi059.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-01T13:30:28.622 INFO:teuthology.orchestra.run.smithi059.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-01T13:30:28.622 INFO:teuthology.orchestra.run.smithi059.stdout: "crimson_alien_op_num_threads": "6", 2024-04-01T13:30:28.622 INFO:teuthology.orchestra.run.smithi059.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-01T13:30:28.622 INFO:teuthology.orchestra.run.smithi059.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-01T13:30:28.622 INFO:teuthology.orchestra.run.smithi059.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-01T13:30:28.622 INFO:teuthology.orchestra.run.smithi059.stdout: "crimson_seastar_cpu_cores": "", 2024-04-01T13:30:28.623 INFO:teuthology.orchestra.run.smithi059.stdout: "crush_location": "", 2024-04-01T13:30:28.623 INFO:teuthology.orchestra.run.smithi059.stdout: "crush_location_hook": "", 2024-04-01T13:30:28.623 INFO:teuthology.orchestra.run.smithi059.stdout: "crush_location_hook_timeout": "10", 2024-04-01T13:30:28.623 INFO:teuthology.orchestra.run.smithi059.stdout: "daemonize": "false", 2024-04-01T13:30:28.623 INFO:teuthology.orchestra.run.smithi059.stdout: "daos_pool": "tank", 2024-04-01T13:30:28.623 INFO:teuthology.orchestra.run.smithi059.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-01T13:30:28.623 INFO:teuthology.orchestra.run.smithi059.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-01T13:30:28.623 INFO:teuthology.orchestra.run.smithi059.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-01T13:30:28.623 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_alienstore": "0/5", 2024-04-01T13:30:28.623 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_allow_any_pool_priority": "false", 2024-04-01T13:30:28.623 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_asok": "1/5", 2024-04-01T13:30:28.623 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_asok_assert_abort": "false", 2024-04-01T13:30:28.623 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_asserts_on_shutdown": "true", 2024-04-01T13:30:28.623 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_auth": "20/20", 2024-04-01T13:30:28.623 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_bdev": "1/3", 2024-04-01T13:30:28.623 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_bluefs": "1/5", 2024-04-01T13:30:28.624 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_bluestore": "1/5", 2024-04-01T13:30:28.624 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_buffer": "0/1", 2024-04-01T13:30:28.624 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_ceph_exporter": "1/5", 2024-04-01T13:30:28.624 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_cephfs_mirror": "0/5", 2024-04-01T13:30:28.624 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_cephsqlite": "0/5", 2024-04-01T13:30:28.624 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_client": "0/5", 2024-04-01T13:30:28.624 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_compressor": "1/5", 2024-04-01T13:30:28.624 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_context": "0/1", 2024-04-01T13:30:28.624 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_crimson_interrupt": "0/5", 2024-04-01T13:30:28.624 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_crush": "1/1", 2024-04-01T13:30:28.624 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_crypto": "1/5", 2024-04-01T13:30:28.624 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_cyanstore": "0/5", 2024-04-01T13:30:28.624 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_deliberately_leak_memory": "false", 2024-04-01T13:30:28.624 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_disable_randomized_ping": "false", 2024-04-01T13:30:28.624 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_dpdk": "1/5", 2024-04-01T13:30:28.625 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_eventtrace": "1/5", 2024-04-01T13:30:28.625 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_filer": "0/1", 2024-04-01T13:30:28.625 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_filestore": "1/3", 2024-04-01T13:30:28.625 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_finisher": "1/1", 2024-04-01T13:30:28.625 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_fuse": "1/5", 2024-04-01T13:30:28.625 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_heartbeat_testing_span": "0", 2024-04-01T13:30:28.625 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_heartbeatmap": "1/5", 2024-04-01T13:30:28.625 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-01T13:30:28.625 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_javaclient": "1/5", 2024-04-01T13:30:28.625 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_journal": "1/3", 2024-04-01T13:30:28.625 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_journaler": "0/5", 2024-04-01T13:30:28.625 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_kstore": "1/5", 2024-04-01T13:30:28.625 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_lockdep": "0/1", 2024-04-01T13:30:28.625 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mclock": "1/5", 2024-04-01T13:30:28.625 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds": "1/5", 2024-04-01T13:30:28.626 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds_balancer": "1/5", 2024-04-01T13:30:28.626 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds_locker": "1/5", 2024-04-01T13:30:28.626 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds_log": "1/5", 2024-04-01T13:30:28.626 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds_log_expire": "1/5", 2024-04-01T13:30:28.626 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds_migrator": "1/5", 2024-04-01T13:30:28.626 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds_quiesce": "3/5", 2024-04-01T13:30:28.626 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_memstore": "1/5", 2024-04-01T13:30:28.626 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mgr": "2/5", 2024-04-01T13:30:28.626 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mgrc": "1/5", 2024-04-01T13:30:28.626 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mon": "20/20", 2024-04-01T13:30:28.626 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_monc": "0/10", 2024-04-01T13:30:28.626 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_ms": "1/1", 2024-04-01T13:30:28.626 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_none": "0/5", 2024-04-01T13:30:28.626 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_objclass": "0/5", 2024-04-01T13:30:28.626 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_objectcacher": "0/5", 2024-04-01T13:30:28.627 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_objecter": "0/1", 2024-04-01T13:30:28.627 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_optracker": "0/5", 2024-04-01T13:30:28.627 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_osd": "1/5", 2024-04-01T13:30:28.627 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_paxos": "20/20", 2024-04-01T13:30:28.627 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_perfcounter": "1/5", 2024-04-01T13:30:28.627 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_prioritycache": "1/5", 2024-04-01T13:30:28.627 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rados": "0/5", 2024-04-01T13:30:28.627 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rbd": "0/5", 2024-04-01T13:30:28.627 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rbd_mirror": "0/5", 2024-04-01T13:30:28.627 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rbd_pwl": "0/5", 2024-04-01T13:30:28.627 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rbd_replay": "0/5", 2024-04-01T13:30:28.627 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_refs": "0/0", 2024-04-01T13:30:28.627 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_reserver": "1/1", 2024-04-01T13:30:28.627 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw": "1/5", 2024-04-01T13:30:28.627 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_access": "1/5", 2024-04-01T13:30:28.627 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_datacache": "1/5", 2024-04-01T13:30:28.628 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_dbstore": "1/5", 2024-04-01T13:30:28.628 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_flight": "1/5", 2024-04-01T13:30:28.628 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-01T13:30:28.628 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_notification": "1/5", 2024-04-01T13:30:28.628 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_sync": "1/5", 2024-04-01T13:30:28.628 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rocksdb": "4/5", 2024-04-01T13:30:28.628 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore": "0/5", 2024-04-01T13:30:28.628 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_backref": "0/5", 2024-04-01T13:30:28.628 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_cache": "0/5", 2024-04-01T13:30:28.628 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_cleaner": "0/5", 2024-04-01T13:30:28.628 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_device": "0/5", 2024-04-01T13:30:28.628 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_epm": "0/5", 2024-04-01T13:30:28.628 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-01T13:30:28.628 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_journal": "0/5", 2024-04-01T13:30:28.628 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_lba": "0/5", 2024-04-01T13:30:28.629 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_odata": "0/5", 2024-04-01T13:30:28.629 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_omap": "0/5", 2024-04-01T13:30:28.629 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_onode": "0/5", 2024-04-01T13:30:28.629 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_t": "0/5", 2024-04-01T13:30:28.629 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_tm": "0/5", 2024-04-01T13:30:28.629 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_striper": "0/1", 2024-04-01T13:30:28.629 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_test": "0/5", 2024-04-01T13:30:28.629 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_throttle": "1/1", 2024-04-01T13:30:28.629 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_timer": "0/1", 2024-04-01T13:30:28.629 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_tp": "0/5", 2024-04-01T13:30:28.629 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_trace": "1/5", 2024-04-01T13:30:28.629 INFO:teuthology.orchestra.run.smithi059.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-01T13:30:28.629 INFO:teuthology.orchestra.run.smithi059.stdout: "device_failure_prediction_mode": "none", 2024-04-01T13:30:28.629 INFO:teuthology.orchestra.run.smithi059.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2024-04-01T13:30:28.629 INFO:teuthology.orchestra.run.smithi059.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-01T13:30:28.630 INFO:teuthology.orchestra.run.smithi059.stdout: "err_to_graylog": "false", 2024-04-01T13:30:28.630 INFO:teuthology.orchestra.run.smithi059.stdout: "err_to_journald": "false", 2024-04-01T13:30:28.630 INFO:teuthology.orchestra.run.smithi059.stdout: "err_to_stderr": "true", 2024-04-01T13:30:28.630 INFO:teuthology.orchestra.run.smithi059.stdout: "err_to_syslog": "false", 2024-04-01T13:30:28.630 INFO:teuthology.orchestra.run.smithi059.stdout: "event_tracing": "false", 2024-04-01T13:30:28.630 INFO:teuthology.orchestra.run.smithi059.stdout: "exporter_addr": "0.0.0.0", 2024-04-01T13:30:28.630 INFO:teuthology.orchestra.run.smithi059.stdout: "exporter_http_port": "9926", 2024-04-01T13:30:28.630 INFO:teuthology.orchestra.run.smithi059.stdout: "exporter_prio_limit": "5", 2024-04-01T13:30:28.630 INFO:teuthology.orchestra.run.smithi059.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-01T13:30:28.630 INFO:teuthology.orchestra.run.smithi059.stdout: "exporter_sort_metrics": "true", 2024-04-01T13:30:28.630 INFO:teuthology.orchestra.run.smithi059.stdout: "exporter_stats_period": "5", 2024-04-01T13:30:28.630 INFO:teuthology.orchestra.run.smithi059.stdout: "fake_statfs_for_testing": "0", 2024-04-01T13:30:28.630 INFO:teuthology.orchestra.run.smithi059.stdout: "fatal_signal_handlers": "true", 2024-04-01T13:30:28.630 INFO:teuthology.orchestra.run.smithi059.stdout: "filer_max_purge_ops": "10", 2024-04-01T13:30:28.631 INFO:teuthology.orchestra.run.smithi059.stdout: "filer_max_truncate_ops": "128", 2024-04-01T13:30:28.631 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_apply_finisher_threads": "1", 2024-04-01T13:30:28.631 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_blackhole": "false", 2024-04-01T13:30:28.631 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_btrfs_clone_range": "true", 2024-04-01T13:30:28.631 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_btrfs_snap": "true", 2024-04-01T13:30:28.631 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_caller_concurrency": "10", 2024-04-01T13:30:28.631 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_collect_device_partition_information": "true", 2024-04-01T13:30:28.631 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_commit_timeout": "600.000000", 2024-04-01T13:30:28.631 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_debug_inject_read_err": "false", 2024-04-01T13:30:28.631 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_debug_omap_check": "false", 2024-04-01T13:30:28.631 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_debug_verify_split": "false", 2024-04-01T13:30:28.631 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_dump_file": "", 2024-04-01T13:30:28.631 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-01T13:30:28.631 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-01T13:30:28.631 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fadvise": "true", 2024-04-01T13:30:28.631 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fail_eio": "true", 2024-04-01T13:30:28.632 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fd_cache_shards": "16", 2024-04-01T13:30:28.632 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fd_cache_size": "128", 2024-04-01T13:30:28.632 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fiemap": "false", 2024-04-01T13:30:28.632 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fiemap_threshold": "4096", 2024-04-01T13:30:28.632 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-01T13:30:28.632 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-01T13:30:28.632 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_inject_stall": "0", 2024-04-01T13:30:28.632 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_journal_parallel": "false", 2024-04-01T13:30:28.632 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_journal_trailing": "false", 2024-04-01T13:30:28.632 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_journal_writeahead": "false", 2024-04-01T13:30:28.632 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_kill_at": "0", 2024-04-01T13:30:28.632 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-01T13:30:28.632 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-01T13:30:28.632 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-01T13:30:28.632 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-01T13:30:28.633 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-01T13:30:28.633 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattrs": "0", 2024-04-01T13:30:28.633 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-01T13:30:28.633 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-01T13:30:28.633 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-01T13:30:28.633 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-01T13:30:28.633 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_xattr_value_size": "0", 2024-04-01T13:30:28.633 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-01T13:30:28.633 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-01T13:30:28.633 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-01T13:30:28.633 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_merge_threshold": "-10", 2024-04-01T13:30:28.633 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-01T13:30:28.633 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_odsync_write": "false", 2024-04-01T13:30:28.633 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_omap_backend": "rocksdb", 2024-04-01T13:30:28.633 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_omap_backend_path": "", 2024-04-01T13:30:28.634 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-01T13:30:28.634 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-01T13:30:28.634 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-01T13:30:28.634 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_op_thread_timeout": "60", 2024-04-01T13:30:28.634 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_op_threads": "2", 2024-04-01T13:30:28.634 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_punch_hole": "false", 2024-04-01T13:30:28.634 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-01T13:30:28.634 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-01T13:30:28.634 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-01T13:30:28.634 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-01T13:30:28.634 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-01T13:30:28.634 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-01T13:30:28.634 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-01T13:30:28.634 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-01T13:30:28.634 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-01T13:30:28.635 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_max_ops": "50", 2024-04-01T13:30:28.635 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-01T13:30:28.635 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_seek_data_hole": "false", 2024-04-01T13:30:28.635 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_sloppy_crc": "false", 2024-04-01T13:30:28.635 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-01T13:30:28.635 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_splice": "false", 2024-04-01T13:30:28.635 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_split_multiple": "2", 2024-04-01T13:30:28.635 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_split_rand_factor": "20", 2024-04-01T13:30:28.635 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_update_to": "1000", 2024-04-01T13:30:28.635 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-01T13:30:28.635 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-01T13:30:28.635 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-01T13:30:28.635 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-01T13:30:28.635 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-01T13:30:28.635 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-01T13:30:28.635 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_enable": "true", 2024-04-01T13:30:28.636 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-01T13:30:28.636 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-01T13:30:28.636 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-01T13:30:28.636 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-01T13:30:28.636 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-01T13:30:28.636 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-01T13:30:28.636 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_xfs_extsize": "false", 2024-04-01T13:30:28.636 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_zfs_snap": "false", 2024-04-01T13:30:28.636 INFO:teuthology.orchestra.run.smithi059.stdout: "fio_dir": "/tmp/fio", 2024-04-01T13:30:28.636 INFO:teuthology.orchestra.run.smithi059.stdout: "fsid": "2a626e12-75d1-4fa9-b59b-fc64723c30fe", 2024-04-01T13:30:28.636 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_allow_other": "true", 2024-04-01T13:30:28.637 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_atomic_o_trunc": "true", 2024-04-01T13:30:28.637 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_big_writes": "true", 2024-04-01T13:30:28.637 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_debug": "false", 2024-04-01T13:30:28.637 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_default_permissions": "false", 2024-04-01T13:30:28.637 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_disable_pagecache": "false", 2024-04-01T13:30:28.637 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_max_write": "0", 2024-04-01T13:30:28.637 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_multithreaded": "true", 2024-04-01T13:30:28.637 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_require_active_mds": "true", 2024-04-01T13:30:28.637 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_set_user_groups": "true", 2024-04-01T13:30:28.637 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_splice_move": "true", 2024-04-01T13:30:28.637 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_splice_read": "true", 2024-04-01T13:30:28.637 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_splice_write": "true", 2024-04-01T13:30:28.637 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-01T13:30:28.637 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_use_invalidate_cb": "true", 2024-04-01T13:30:28.638 INFO:teuthology.orchestra.run.smithi059.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-01T13:30:28.638 INFO:teuthology.orchestra.run.smithi059.stdout: "gss_target_name": "ceph", 2024-04-01T13:30:28.638 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_file": "", 2024-04-01T13:30:28.638 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_inject_failure": "0", 2024-04-01T13:30:28.638 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_interval": "5", 2024-04-01T13:30:28.638 INFO:teuthology.orchestra.run.smithi059.stdout: "host": "smithi059", 2024-04-01T13:30:28.638 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-01T13:30:28.638 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-01T13:30:28.638 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-01T13:30:28.638 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-01T13:30:28.638 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-01T13:30:28.638 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-01T13:30:28.639 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-01T13:30:28.639 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-01T13:30:28.639 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-01T13:30:28.639 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-01T13:30:28.639 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-01T13:30:28.639 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-01T13:30:28.639 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-01T13:30:28.639 INFO:teuthology.orchestra.run.smithi059.stdout: "inject_early_sigterm": "false", 2024-04-01T13:30:28.639 INFO:teuthology.orchestra.run.smithi059.stdout: "jaeger_agent_port": "6799", 2024-04-01T13:30:28.639 INFO:teuthology.orchestra.run.smithi059.stdout: "jaeger_tracing_enable": "false", 2024-04-01T13:30:28.639 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_aio": "true", 2024-04-01T13:30:28.639 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_align_min_size": "65536", 2024-04-01T13:30:28.639 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_block_align": "true", 2024-04-01T13:30:28.639 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_block_size": "4096", 2024-04-01T13:30:28.640 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_dio": "true", 2024-04-01T13:30:28.640 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_discard": "false", 2024-04-01T13:30:28.640 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_force_aio": "false", 2024-04-01T13:30:28.640 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_ignore_corruption": "false", 2024-04-01T13:30:28.640 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_max_write_bytes": "10485760", 2024-04-01T13:30:28.640 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_max_write_entries": "100", 2024-04-01T13:30:28.640 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_replay_from": "0", 2024-04-01T13:30:28.640 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-01T13:30:28.640 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-01T13:30:28.640 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-01T13:30:28.640 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-01T13:30:28.640 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_write_header_frequency": "0", 2024-04-01T13:30:28.640 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_zero_on_create": "false", 2024-04-01T13:30:28.640 INFO:teuthology.orchestra.run.smithi059.stdout: "journaler_prefetch_periods": "10", 2024-04-01T13:30:28.640 INFO:teuthology.orchestra.run.smithi059.stdout: "journaler_prezero_periods": "5", 2024-04-01T13:30:28.641 INFO:teuthology.orchestra.run.smithi059.stdout: "journaler_write_head_interval": "15", 2024-04-01T13:30:28.641 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "", 2024-04-01T13:30:28.641 INFO:teuthology.orchestra.run.smithi059.stdout: "keyfile": "", 2024-04-01T13:30:28.641 INFO:teuthology.orchestra.run.smithi059.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-01T13:30:28.641 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_backend": "rocksdb", 2024-04-01T13:30:28.641 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_default_stripe_size": "65536", 2024-04-01T13:30:28.641 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_fsck_on_mount": "false", 2024-04-01T13:30:28.641 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-01T13:30:28.641 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_max_bytes": "67108864", 2024-04-01T13:30:28.641 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_max_ops": "512", 2024-04-01T13:30:28.641 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_nid_prealloc": "1024", 2024-04-01T13:30:28.641 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_onode_map_size": "1024", 2024-04-01T13:30:28.641 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-01T13:30:28.641 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_sync_submit_transaction": "false", 2024-04-01T13:30:28.641 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_sync_transaction": "false", 2024-04-01T13:30:28.642 INFO:teuthology.orchestra.run.smithi059.stdout: "librados_thread_count": "2", 2024-04-01T13:30:28.642 INFO:teuthology.orchestra.run.smithi059.stdout: "lockdep": "false", 2024-04-01T13:30:28.642 INFO:teuthology.orchestra.run.smithi059.stdout: "lockdep_force_backtrace": "false", 2024-04-01T13:30:28.642 INFO:teuthology.orchestra.run.smithi059.stdout: "log_coarse_timestamps": "true", 2024-04-01T13:30:28.642 INFO:teuthology.orchestra.run.smithi059.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-01T13:30:28.642 INFO:teuthology.orchestra.run.smithi059.stdout: "log_flush_on_exit": "false", 2024-04-01T13:30:28.642 INFO:teuthology.orchestra.run.smithi059.stdout: "log_graylog_host": "127.0.0.1", 2024-04-01T13:30:28.642 INFO:teuthology.orchestra.run.smithi059.stdout: "log_graylog_port": "12201", 2024-04-01T13:30:28.642 INFO:teuthology.orchestra.run.smithi059.stdout: "log_max_new": "1000", 2024-04-01T13:30:28.642 INFO:teuthology.orchestra.run.smithi059.stdout: "log_max_recent": "10000", 2024-04-01T13:30:28.642 INFO:teuthology.orchestra.run.smithi059.stdout: "log_stderr_prefix": "", 2024-04-01T13:30:28.642 INFO:teuthology.orchestra.run.smithi059.stdout: "log_stop_at_utilization": "0.970000", 2024-04-01T13:30:28.642 INFO:teuthology.orchestra.run.smithi059.stdout: "log_to_file": "true", 2024-04-01T13:30:28.642 INFO:teuthology.orchestra.run.smithi059.stdout: "log_to_graylog": "false", 2024-04-01T13:30:28.643 INFO:teuthology.orchestra.run.smithi059.stdout: "log_to_journald": "false", 2024-04-01T13:30:28.643 INFO:teuthology.orchestra.run.smithi059.stdout: "log_to_stderr": "false", 2024-04-01T13:30:28.643 INFO:teuthology.orchestra.run.smithi059.stdout: "log_to_syslog": "false", 2024-04-01T13:30:28.643 INFO:teuthology.orchestra.run.smithi059.stdout: "max_rotating_auth_attempts": "10", 2024-04-01T13:30:28.643 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-01T13:30:28.643 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_action_on_write_error": "1", 2024-04-01T13:30:28.643 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_alternate_name_max": "8192", 2024-04-01T13:30:28.643 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_asio_thread_count": "2", 2024-04-01T13:30:28.643 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_export_pin": "true", 2024-04-01T13:30:28.643 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_fragment_dirs": "true", 2024-04-01T13:30:28.643 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-01T13:30:28.643 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_fragment_interval": "5", 2024-04-01T13:30:28.643 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-01T13:30:28.643 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-01T13:30:28.643 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_interval": "10", 2024-04-01T13:30:28.644 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_max": "-1", 2024-04-01T13:30:28.644 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_max_until": "-1", 2024-04-01T13:30:28.644 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_merge_size": "50", 2024-04-01T13:30:28.644 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_midchunk": "0.300000", 2024-04-01T13:30:28.644 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-01T13:30:28.644 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_min_start": "0.200000", 2024-04-01T13:30:28.644 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_minchunk": "0.001000", 2024-04-01T13:30:28.644 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_mode": "0", 2024-04-01T13:30:28.644 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_need_max": "1.200000", 2024-04-01T13:30:28.644 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_need_min": "0.800000", 2024-04-01T13:30:28.644 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_overload_epochs": "2", 2024-04-01T13:30:28.644 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-01T13:30:28.644 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-01T13:30:28.644 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_split_bits": "3", 2024-04-01T13:30:28.645 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-01T13:30:28.645 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_split_size": "10000", 2024-04-01T13:30:28.645 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-01T13:30:28.645 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_target_decay": "10.000000", 2024-04-01T13:30:28.645 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-01T13:30:28.645 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_beacon_grace": "15.000000", 2024-04-01T13:30:28.645 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_beacon_interval": "4.000000", 2024-04-01T13:30:28.645 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-01T13:30:28.645 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-01T13:30:28.645 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_mid": "0.700000", 2024-04-01T13:30:28.645 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-01T13:30:28.645 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_quiesce_delay": "0", 2024-04-01T13:30:28.645 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-01T13:30:28.645 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-01T13:30:28.645 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-01T13:30:28.645 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_release_free_interval": "10", 2024-04-01T13:30:28.646 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_reservation": "0.050000", 2024-04-01T13:30:28.646 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-01T13:30:28.646 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_trim_interval": "1", 2024-04-01T13:30:28.646 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_trim_threshold": "262144", 2024-04-01T13:30:28.646 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-01T13:30:28.646 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-01T13:30:28.646 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-01T13:30:28.646 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_client_prealloc_inos": "1000", 2024-04-01T13:30:28.646 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-01T13:30:28.646 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_connect_bootstrapping": "false", 2024-04-01T13:30:28.646 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_damage_table_max_entries": "10000", 2024-04-01T13:30:28.646 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-01T13:30:28.646 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_debug_auth_pins": "false", 2024-04-01T13:30:28.646 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_debug_frag": "false", 2024-04-01T13:30:28.646 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_debug_scatterstat": "false", 2024-04-01T13:30:28.647 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_debug_subtrees": "false", 2024-04-01T13:30:28.647 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_decay_halflife": "5.000000", 2024-04-01T13:30:28.647 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_default_dir_hash": "2", 2024-04-01T13:30:28.647 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_defer_session_stale": "true", 2024-04-01T13:30:28.647 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_deny_all_reconnect": "false", 2024-04-01T13:30:28.647 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dir_keys_per_op": "16384", 2024-04-01T13:30:28.647 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dir_max_commit_size": "10", 2024-04-01T13:30:28.647 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dir_max_entries": "0", 2024-04-01T13:30:28.647 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dir_prefetch": "true", 2024-04-01T13:30:28.647 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-01T13:30:28.647 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-01T13:30:28.647 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dump_cache_on_map": "false", 2024-04-01T13:30:28.647 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-01T13:30:28.647 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-01T13:30:28.648 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_early_reply": "true", 2024-04-01T13:30:28.648 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_enable_op_tracker": "true", 2024-04-01T13:30:28.648 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_enforce_unique_name": "true", 2024-04-01T13:30:28.648 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-01T13:30:28.648 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-01T13:30:28.648 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_export_ephemeral_random": "true", 2024-04-01T13:30:28.648 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-01T13:30:28.648 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-01T13:30:28.648 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-01T13:30:28.648 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-01T13:30:28.648 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-01T13:30:28.648 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-01T13:30:28.648 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-01T13:30:28.648 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-01T13:30:28.648 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_health_summarize_threshold": "10", 2024-04-01T13:30:28.648 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-01T13:30:28.649 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-01T13:30:28.649 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_inject_health_dummy": "false", 2024-04-01T13:30:28.649 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-01T13:30:28.649 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_inject_migrator_session_race": "false", 2024-04-01T13:30:28.649 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-01T13:30:28.649 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-01T13:30:28.649 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-01T13:30:28.649 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_join_fs": "", 2024-04-01T13:30:28.649 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_journal_format": "1", 2024-04-01T13:30:28.649 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_create_at": "0", 2024-04-01T13:30:28.649 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_export_at": "0", 2024-04-01T13:30:28.649 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_import_at": "0", 2024-04-01T13:30:28.649 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_journal_at": "0", 2024-04-01T13:30:28.649 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_journal_expire_at": "0", 2024-04-01T13:30:28.650 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_journal_replay_at": "0", 2024-04-01T13:30:28.650 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_link_at": "0", 2024-04-01T13:30:28.650 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_mdstable_at": "0", 2024-04-01T13:30:28.650 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_openc_at": "0", 2024-04-01T13:30:28.650 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_rename_at": "0", 2024-04-01T13:30:28.650 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_shutdown_at": "0", 2024-04-01T13:30:28.650 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-01T13:30:28.650 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_event_large_threshold": "524288", 2024-04-01T13:30:28.650 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_events_per_segment": "1024", 2024-04-01T13:30:28.650 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-01T13:30:28.650 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_max_events": "-1", 2024-04-01T13:30:28.650 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_max_segments": "128", 2024-04-01T13:30:28.650 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_pause": "false", 2024-04-01T13:30:28.650 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_segment_size": "0", 2024-04-01T13:30:28.650 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-01T13:30:28.651 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-01T13:30:28.651 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-01T13:30:28.651 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_trim_threshold": "128", 2024-04-01T13:30:28.651 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-01T13:30:28.651 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_warn_factor": "2.000000", 2024-04-01T13:30:28.651 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_caps_per_client": "1048576", 2024-04-01T13:30:28.651 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_completed_flushes": "100000", 2024-04-01T13:30:28.651 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_completed_requests": "100000", 2024-04-01T13:30:28.651 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_export_size": "20971520", 2024-04-01T13:30:28.651 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_file_recover": "32", 2024-04-01T13:30:28.651 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_purge_files": "64", 2024-04-01T13:30:28.651 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_purge_ops": "8192", 2024-04-01T13:30:28.651 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-01T13:30:28.651 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-01T13:30:28.651 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_snaps_per_dir": "100", 2024-04-01T13:30:28.651 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_metrics_update_interval": "2", 2024-04-01T13:30:28.652 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_min_caps_per_client": "100", 2024-04-01T13:30:28.652 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_min_caps_working_set": "10000", 2024-04-01T13:30:28.652 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-01T13:30:28.652 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_numa_node": "-1", 2024-04-01T13:30:28.652 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-01T13:30:28.652 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_op_complaint_time": "30.000000", 2024-04-01T13:30:28.652 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_op_history_duration": "600", 2024-04-01T13:30:28.652 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_op_history_size": "20", 2024-04-01T13:30:28.652 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_op_history_slow_op_size": "20", 2024-04-01T13:30:28.652 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-01T13:30:28.652 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_op_log_threshold": "5", 2024-04-01T13:30:28.652 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_ping_grace": "15", 2024-04-01T13:30:28.652 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_ping_interval": "5", 2024-04-01T13:30:28.652 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-01T13:30:28.652 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-01T13:30:28.653 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_recall_max_caps": "30000", 2024-04-01T13:30:28.653 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-01T13:30:28.653 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-01T13:30:28.653 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-01T13:30:28.653 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_recall_warning_threshold": "262144", 2024-04-01T13:30:28.653 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-01T13:30:28.653 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_replay_interval": "1.000000", 2024-04-01T13:30:28.653 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-01T13:30:28.653 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-01T13:30:28.653 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_root_ino_gid": "0", 2024-04-01T13:30:28.653 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_root_ino_uid": "0", 2024-04-01T13:30:28.653 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-01T13:30:28.653 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-01T13:30:28.653 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-01T13:30:28.653 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-01T13:30:28.654 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-01T13:30:28.654 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-01T13:30:28.654 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-01T13:30:28.654 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-01T13:30:28.654 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-01T13:30:28.654 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-01T13:30:28.654 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_shutdown_check": "0", 2024-04-01T13:30:28.654 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_skip_ino": "0", 2024-04-01T13:30:28.654 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-01T13:30:28.654 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_snap_max_uid": "4294967294", 2024-04-01T13:30:28.654 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_snap_min_uid": "0", 2024-04-01T13:30:28.654 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_snap_rstat": "false", 2024-04-01T13:30:28.654 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_standby_replay_damaged": "false", 2024-04-01T13:30:28.654 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_symlink_recovery": "true", 2024-04-01T13:30:28.654 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-01T13:30:28.655 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_thrash_exports": "0", 2024-04-01T13:30:28.655 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_thrash_fragments": "0", 2024-04-01T13:30:28.655 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_tick_interval": "5.000000", 2024-04-01T13:30:28.655 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_valgrind_exit": "false", 2024-04-01T13:30:28.655 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_verify_backtrace": "1", 2024-04-01T13:30:28.655 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_verify_scatter": "false", 2024-04-01T13:30:28.655 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-01T13:30:28.655 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_wipe_sessions": "false", 2024-04-01T13:30:28.655 INFO:teuthology.orchestra.run.smithi059.stdout: "mempool_debug": "false", 2024-04-01T13:30:28.655 INFO:teuthology.orchestra.run.smithi059.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-01T13:30:28.655 INFO:teuthology.orchestra.run.smithi059.stdout: "memstore_device_bytes": "1073741824", 2024-04-01T13:30:28.655 INFO:teuthology.orchestra.run.smithi059.stdout: "memstore_page_set": "false", 2024-04-01T13:30:28.655 INFO:teuthology.orchestra.run.smithi059.stdout: "memstore_page_size": "65536", 2024-04-01T13:30:28.655 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_client_bytes": "134217728", 2024-04-01T13:30:28.656 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_client_messages": "512", 2024-04-01T13:30:28.656 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-01T13:30:28.656 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-01T13:30:28.656 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-01T13:30:28.656 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-01T13:30:28.656 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_disabled_modules": "", 2024-04-01T13:30:28.656 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-01T13:30:28.656 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_max_pg_num_change": "128", 2024-04-01T13:30:28.656 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_mds_bytes": "134217728", 2024-04-01T13:30:28.656 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_mds_messages": "128", 2024-04-01T13:30:28.656 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-01T13:30:28.656 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_mon_bytes": "134217728", 2024-04-01T13:30:28.656 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_mon_messages": "128", 2024-04-01T13:30:28.656 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_osd_bytes": "536870912", 2024-04-01T13:30:28.656 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_osd_messages": "8192", 2024-04-01T13:30:28.657 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_pool": "true", 2024-04-01T13:30:28.657 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-01T13:30:28.657 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_standby_modules": "true", 2024-04-01T13:30:28.657 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_stats_period": "5", 2024-04-01T13:30:28.657 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_stats_threshold": "5", 2024-04-01T13:30:28.657 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_tick_period": "2", 2024-04-01T13:30:28.657 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-01T13:30:28.657 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-01T13:30:28.657 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_allow_pool_delete": "true", 2024-04-01T13:30:28.657 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_allow_pool_size_one": "true", 2024-04-01T13:30:28.657 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_auth_validate_all_caps": "true", 2024-04-01T13:30:28.657 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-01T13:30:28.657 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-01T13:30:28.657 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_bytes": "104857600", 2024-04-01T13:30:28.657 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_directed_command_retry": "2", 2024-04-01T13:30:28.658 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-01T13:30:28.658 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-01T13:30:28.658 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-01T13:30:28.658 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-01T13:30:28.658 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_hunt_parallel": "3", 2024-04-01T13:30:28.658 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_log_interval": "1.000000", 2024-04-01T13:30:28.658 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-01T13:30:28.658 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_ping_interval": "10.000000", 2024-04-01T13:30:28.658 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-01T13:30:28.658 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_target_rank": "-1", 2024-04-01T13:30:28.658 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-01T13:30:28.658 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-01T13:30:28.658 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-01T13:30:28.658 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_file_level": "debug", 2024-04-01T13:30:28.659 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_file": "true", 2024-04-01T13:30:28.659 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-01T13:30:28.659 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-01T13:30:28.659 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-01T13:30:28.659 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_journald": "false", 2024-04-01T13:30:28.659 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-01T13:30:28.659 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-01T13:30:28.659 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-01T13:30:28.659 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-04-01T13:30:28.659 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_compact_on_bootstrap": "false", 2024-04-01T13:30:28.659 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_compact_on_start": "false", 2024-04-01T13:30:28.659 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_compact_on_trim": "true", 2024-04-01T13:30:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-01T13:30:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-01T13:30:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-01T13:30:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cpu_threads": "4", 2024-04-01T13:30:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_crush_min_required_version": "hammer", 2024-04-01T13:30:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_daemon_bytes": "419430400", 2024-04-01T13:30:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-01T13:30:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_data_avail_crit": "5", 2024-04-01T13:30:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_data_avail_warn": "5", 2024-04-01T13:30:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_data_size_warn": "16106127360", 2024-04-01T13:30:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-01T13:30:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-01T13:30:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_dump_json": "false", 2024-04-01T13:30:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-01T13:30:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_dump_transactions": "false", 2024-04-01T13:30:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_extra_checks": "false", 2024-04-01T13:30:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-01T13:30:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-01T13:30:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_no_require_reef": "false", 2024-04-01T13:30:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_no_require_squid": "false", 2024-04-01T13:30:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-01T13:30:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-01T13:30:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-01T13:30:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_down_mkfs_grace": "120", 2024-04-01T13:30:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_down_uptime_grace": "60", 2024-04-01T13:30:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_election_default_strategy": "1", 2024-04-01T13:30:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_election_timeout": "5.000000", 2024-04-01T13:30:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-01T13:30:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_elector_ping_divisor": "2", 2024-04-01T13:30:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-01T13:30:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_enable_op_tracker": "true", 2024-04-01T13:30:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_fake_pool_delete": "false", 2024-04-01T13:30:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_force_quorum_join": "false", 2024-04-01T13:30:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_globalid_prealloc": "10000", 2024-04-01T13:30:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_health_detail_to_clog": "false", 2024-04-01T13:30:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_health_log_update_period": "5", 2024-04-01T13:30:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_health_max_detail": "50", 2024-04-01T13:30:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_health_to_clog": "true", 2024-04-01T13:30:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_health_to_clog_interval": "600", 2024-04-01T13:30:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-01T13:30:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_host": "172.21.15.59", 2024-04-01T13:30:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_host_override": "", 2024-04-01T13:30:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_initial_members": "", 2024-04-01T13:30:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-01T13:30:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-01T13:30:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-01T13:30:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-01T13:30:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-01T13:30:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_lease": "5.000000", 2024-04-01T13:30:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-01T13:30:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-01T13:30:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_log_full_interval": "50", 2024-04-01T13:30:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_log_max": "10000", 2024-04-01T13:30:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_log_max_summary": "50", 2024-04-01T13:30:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-01T13:30:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_log_epochs": "500", 2024-04-01T13:30:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-01T13:30:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-01T13:30:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_osd": "10000", 2024-04-01T13:30:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_pg_per_osd": "10000", 2024-04-01T13:30:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_pool_pg_num": "65536", 2024-04-01T13:30:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-01T13:30:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-01T13:30:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mds_force_trim_to": "0", 2024-04-01T13:30:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mds_skip_sanity": "false", 2024-04-01T13:30:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_memory_autotune": "true", 2024-04-01T13:30:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_memory_target": "2147483648", 2024-04-01T13:30:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mgr_beacon_grace": "30", 2024-04-01T13:30:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-01T13:30:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mgr_digest_period": "5", 2024-04-01T13:30:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mgr_inactive_grace": "60", 2024-04-01T13:30:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-01T13:30:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-01T13:30:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_min_osdmap_epochs": "50", 2024-04-01T13:30:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_op_complaint_time": "30", 2024-04-01T13:30:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_op_history_duration": "600", 2024-04-01T13:30:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_op_history_size": "20", 2024-04-01T13:30:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_op_history_slow_op_size": "20", 2024-04-01T13:30:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-01T13:30:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_op_log_threshold": "5", 2024-04-01T13:30:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-01T13:30:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-01T13:30:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-01T13:30:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_auto_mark_in": "false", 2024-04-01T13:30:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-01T13:30:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-01T13:30:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-01T13:30:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_cache_size": "500", 2024-04-01T13:30:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-01T13:30:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-01T13:30:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-01T13:30:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_down_out_interval": "600", 2024-04-01T13:30:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-01T13:30:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-01T13:30:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_force_trim_to": "0", 2024-04-01T13:30:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-01T13:30:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-01T13:30:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-01T13:30:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-01T13:30:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-01T13:30:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-01T13:30:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_max_creating_pgs": "1024", 2024-04-01T13:30:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-01T13:30:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_min_down_reporters": "2", 2024-04-01T13:30:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-01T13:30:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-01T13:30:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-01T13:30:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-01T13:30:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-01T13:30:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-01T13:30:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_report_timeout": "900", 2024-04-01T13:30:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-01T13:30:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-01T13:30:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-01T13:30:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-01T13:30:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-01T13:30:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-01T13:30:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-01T13:30:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-01T13:30:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-01T13:30:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pg_stuck_threshold": "60", 2024-04-01T13:30:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-01T13:30:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-01T13:30:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-01T13:30:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-01T13:30:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-01T13:30:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-01T13:30:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_probe_timeout": "2.000000", 2024-04-01T13:30:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_reweight_max_change": "0.050000", 2024-04-01T13:30:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_reweight_max_osds": "4", 2024-04-01T13:30:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-01T13:30:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-01T13:30:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-01T13:30:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-01T13:30:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-01T13:30:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_scrub_interval": "86400", 2024-04-01T13:30:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_scrub_max_keys": "100", 2024-04-01T13:30:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_scrub_timeout": "300", 2024-04-01T13:30:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_session_timeout": "300", 2024-04-01T13:30:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_smart_report_timeout": "5", 2024-04-01T13:30:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_stat_smooth_intervals": "6", 2024-04-01T13:30:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-01T13:30:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_stretch_pool_min_size": "2", 2024-04-01T13:30:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_stretch_pool_size": "4", 2024-04-01T13:30:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-01T13:30:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-01T13:30:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_sync_debug": "false", 2024-04-01T13:30:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-01T13:30:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-01T13:30:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_sync_provider_kill_at": "0", 2024-04-01T13:30:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_sync_requester_kill_at": "0", 2024-04-01T13:30:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_sync_timeout": "60.000000", 2024-04-01T13:30:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_target_pg_per_osd": "100", 2024-04-01T13:30:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_tick_interval": "5", 2024-04-01T13:30:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_timecheck_interval": "300.000000", 2024-04-01T13:30:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-01T13:30:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_use_min_delay_socket": "false", 2024-04-01T13:30:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_older_version_delay": "604800", 2024-04-01T13:30:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-01T13:30:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-01T13:30:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-01T13:30:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-01T13:30:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-01T13:30:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-01T13:30:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-01T13:30:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_misplaced": "false", 2024-04-01T13:30:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-01T13:30:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_older_version": "true", 2024-04-01T13:30:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-01T13:30:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-01T13:30:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-01T13:30:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-01T13:30:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-01T13:30:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-01T13:30:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-01T13:30:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-01T13:30:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-01T13:30:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: "monmap": "", 2024-04-01T13:30:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-01T13:30:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-01T13:30:28.672 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-01T13:30:28.672 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-01T13:30:28.672 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-01T13:30:28.672 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-01T13:30:28.672 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_tracing_enabled": "false", 2024-04-01T13:30:28.672 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_op_threads": "3", 2024-04-01T13:30:28.672 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-01T13:30:28.672 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_cm": "false", 2024-04-01T13:30:28.672 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_device_name": "", 2024-04-01T13:30:28.672 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_dscp": "96", 2024-04-01T13:30:28.672 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-01T13:30:28.672 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-01T13:30:28.672 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_local_gid": "", 2024-04-01T13:30:28.672 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-01T13:30:28.672 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_port_num": "1", 2024-04-01T13:30:28.673 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-01T13:30:28.673 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-01T13:30:28.673 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-01T13:30:28.673 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-01T13:30:28.673 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_sl": "3", 2024-04-01T13:30:28.673 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_support_srq": "true", 2024-04-01T13:30:28.673 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_type": "ib", 2024-04-01T13:30:28.673 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_reap_threshold": "5", 2024-04-01T13:30:28.673 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_before_connect": "false", 2024-04-01T13:30:28.673 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_ipv4": "true", 2024-04-01T13:30:28.673 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_ipv6": "false", 2024-04-01T13:30:28.673 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_msgr1": "true", 2024-04-01T13:30:28.673 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_msgr2": "true", 2024-04-01T13:30:28.673 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_port_max": "7568", 2024-04-01T13:30:28.673 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_port_min": "6800", 2024-04-01T13:30:28.674 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-01T13:30:28.674 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_retry_count": "3", 2024-04-01T13:30:28.674 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_retry_delay": "5", 2024-04-01T13:30:28.674 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_blackhole_client": "false", 2024-04-01T13:30:28.674 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_blackhole_mds": "false", 2024-04-01T13:30:28.674 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_blackhole_mgr": "false", 2024-04-01T13:30:28.674 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_blackhole_mon": "false", 2024-04-01T13:30:28.674 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_blackhole_osd": "false", 2024-04-01T13:30:28.674 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_client_mode": "crc", 2024-04-01T13:30:28.674 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_cluster_mode": "crc", 2024-04-01T13:30:28.674 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_cluster_type": "", 2024-04-01T13:30:28.674 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_compress_secure": "false", 2024-04-01T13:30:28.674 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_connection_idle_timeout": "900", 2024-04-01T13:30:28.674 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_connection_ready_timeout": "10", 2024-04-01T13:30:28.675 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_crc_data": "true", 2024-04-01T13:30:28.675 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_crc_header": "true", 2024-04-01T13:30:28.675 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_die_on_bad_msg": "false", 2024-04-01T13:30:28.675 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_die_on_bug": "true", 2024-04-01T13:30:28.675 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_die_on_old_message": "true", 2024-04-01T13:30:28.675 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_die_on_skipped_message": "false", 2024-04-01T13:30:28.675 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-01T13:30:28.675 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-01T13:30:28.675 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_coremask": "0xF", 2024-04-01T13:30:28.675 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-01T13:30:28.675 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-01T13:30:28.675 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_enable_tso": "true", 2024-04-01T13:30:28.675 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-01T13:30:28.675 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-01T13:30:28.675 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_hugepages": "", 2024-04-01T13:30:28.675 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-01T13:30:28.676 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-01T13:30:28.676 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_lro": "true", 2024-04-01T13:30:28.676 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_memory_channel": "4", 2024-04-01T13:30:28.676 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-01T13:30:28.676 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_pmd": "", 2024-04-01T13:30:28.676 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_port_id": "0", 2024-04-01T13:30:28.676 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-01T13:30:28.676 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-01T13:30:28.676 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dump_on_send": "false", 2024-04-01T13:30:28.676 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_initial_backoff": "0.200000", 2024-04-01T13:30:28.676 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_inject_delay_max": "1.000000", 2024-04-01T13:30:28.676 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-01T13:30:28.676 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_inject_delay_type": "", 2024-04-01T13:30:28.676 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-01T13:30:28.676 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_inject_network_congestion": "0", 2024-04-01T13:30:28.677 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_inject_socket_failures": "0", 2024-04-01T13:30:28.677 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_learn_addr_from_peer": "true", 2024-04-01T13:30:28.677 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_max_accept_failures": "4", 2024-04-01T13:30:28.677 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_max_backoff": "15.000000", 2024-04-01T13:30:28.677 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_mon_client_mode": "crc", 2024-04-01T13:30:28.677 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_mon_cluster_mode": "crc", 2024-04-01T13:30:28.677 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_mon_service_mode": "crc", 2024-04-01T13:30:28.677 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_osd_compress_min_size": "1024", 2024-04-01T13:30:28.677 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_osd_compress_mode": "none", 2024-04-01T13:30:28.677 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-01T13:30:28.677 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-01T13:30:28.677 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_pq_min_cost": "65536", 2024-04-01T13:30:28.677 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_public_type": "", 2024-04-01T13:30:28.677 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_service_mode": "crc", 2024-04-01T13:30:28.678 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_tcp_listen_backlog": "512", 2024-04-01T13:30:28.678 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_tcp_nodelay": "true", 2024-04-01T13:30:28.678 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-01T13:30:28.678 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_tcp_rcvbuf": "0", 2024-04-01T13:30:28.678 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_type": "async+posix", 2024-04-01T13:30:28.678 INFO:teuthology.orchestra.run.smithi059.stdout: "no_config_file": "false", 2024-04-01T13:30:28.678 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_completion_locks_per_session": "32", 2024-04-01T13:30:28.678 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-01T13:30:28.678 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-01T13:30:28.678 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_inflight_ops": "1024", 2024-04-01T13:30:28.678 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-01T13:30:28.678 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-01T13:30:28.678 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_tick_interval": "5.000000", 2024-04-01T13:30:28.678 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_timeout": "10.000000", 2024-04-01T13:30:28.678 INFO:teuthology.orchestra.run.smithi059.stdout: "objectstore_blackhole": "false", 2024-04-01T13:30:28.679 INFO:teuthology.orchestra.run.smithi059.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-01T13:30:28.679 INFO:teuthology.orchestra.run.smithi059.stdout: "openssl_engine_opts": "", 2024-04-01T13:30:28.679 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_delay_time": "5.000000", 2024-04-01T13:30:28.679 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_hist_halflife": "1000", 2024-04-01T13:30:28.679 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_max_low_ops": "2", 2024-04-01T13:30:28.679 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_max_ops": "4", 2024-04-01T13:30:28.679 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-01T13:30:28.679 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-01T13:30:28.679 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_slop": "0.020000", 2024-04-01T13:30:28.679 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-01T13:30:28.679 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-01T13:30:28.679 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_asio_thread_count": "2", 2024-04-01T13:30:28.679 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_async_recovery_min_cost": "100", 2024-04-01T13:30:28.679 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-01T13:30:28.679 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-01T13:30:28.680 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_backfill_scan_max": "512", 2024-04-01T13:30:28.680 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_backfill_scan_min": "64", 2024-04-01T13:30:28.680 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_backoff_on_degraded": "false", 2024-04-01T13:30:28.680 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_backoff_on_peering": "false", 2024-04-01T13:30:28.680 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_backoff_on_unfound": "true", 2024-04-01T13:30:28.680 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_beacon_report_interval": "300", 2024-04-01T13:30:28.680 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_bench_duration": "30", 2024-04-01T13:30:28.680 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-01T13:30:28.680 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_bench_max_block_size": "67108864", 2024-04-01T13:30:28.680 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-01T13:30:28.680 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_blkin_trace_all": "false", 2024-04-01T13:30:28.681 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-01T13:30:28.681 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-01T13:30:28.681 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-01T13:30:28.681 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-01T13:30:28.681 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_check_for_log_corruption": "false", 2024-04-01T13:30:28.681 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-01T13:30:28.681 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-01T13:30:28.681 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-01T13:30:28.681 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-01T13:30:28.681 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_class_update_on_start": "true", 2024-04-01T13:30:28.681 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_client_message_cap": "256", 2024-04-01T13:30:28.681 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_client_message_size_cap": "524288000", 2024-04-01T13:30:28.681 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_client_op_priority": "63", 2024-04-01T13:30:28.681 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_client_watch_timeout": "30", 2024-04-01T13:30:28.681 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_command_max_records": "256", 2024-04-01T13:30:28.682 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_compact_on_start": "false", 2024-04-01T13:30:28.682 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-01T13:30:28.682 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-01T13:30:28.682 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-01T13:30:28.682 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_crush_update_on_start": "true", 2024-04-01T13:30:28.682 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_crush_update_weight_set": "true", 2024-04-01T13:30:28.682 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-01T13:30:28.682 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-01T13:30:28.682 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-01T13:30:28.682 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-01T13:30:28.682 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-01T13:30:28.682 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_feed_pullee": "-1", 2024-04-01T13:30:28.682 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-01T13:30:28.682 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-01T13:30:28.683 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-01T13:30:28.683 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_misdirected_ops": "false", 2024-04-01T13:30:28.683 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_no_acting_change": "false", 2024-04-01T13:30:28.683 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_no_purge_strays": "false", 2024-04-01T13:30:28.683 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_op_order": "false", 2024-04-01T13:30:28.683 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-01T13:30:28.683 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-01T13:30:28.683 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-01T13:30:28.683 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-01T13:30:28.683 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_shutdown": "false", 2024-04-01T13:30:28.683 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-01T13:30:28.683 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-01T13:30:28.683 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_trim_objects": "false", 2024-04-01T13:30:28.683 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-01T13:30:28.683 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-01T13:30:28.684 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_verify_snaps": "false", 2024-04-01T13:30:28.684 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-01T13:30:28.684 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-01T13:30:28.684 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_keys": "1024", 2024-04-01T13:30:28.684 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-01T13:30:28.684 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-01T13:30:28.684 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-01T13:30:28.684 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_stride": "524288", 2024-04-01T13:30:28.684 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-01T13:30:28.684 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-01T13:30:28.684 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_default_notify_timeout": "30", 2024-04-01T13:30:28.684 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_delete_sleep": "0.000000", 2024-04-01T13:30:28.684 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-01T13:30:28.684 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-01T13:30:28.684 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-01T13:30:28.685 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_discard_disconnected_ops": "true", 2024-04-01T13:30:28.685 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_enable_op_tracker": "true", 2024-04-01T13:30:28.685 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-01T13:30:28.685 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-01T13:30:28.685 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-01T13:30:28.685 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-01T13:30:28.685 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_fast_info": "true", 2024-04-01T13:30:28.685 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_fast_shutdown": "true", 2024-04-01T13:30:28.685 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-01T13:30:28.685 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-01T13:30:28.685 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-01T13:30:28.685 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-01T13:30:28.685 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-01T13:30:28.685 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_function_tracing": "false", 2024-04-01T13:30:28.685 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_grace": "20", 2024-04-01T13:30:28.686 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_interval": "6", 2024-04-01T13:30:28.686 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-01T13:30:28.686 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_min_peers": "10", 2024-04-01T13:30:28.686 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_min_size": "2000", 2024-04-01T13:30:28.686 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_stale": "600", 2024-04-01T13:30:28.686 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-01T13:30:28.686 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_hit_set_max_size": "100000", 2024-04-01T13:30:28.686 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_hit_set_min_size": "1000", 2024-04-01T13:30:28.686 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-01T13:30:28.686 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-01T13:30:28.686 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-01T13:30:28.686 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-01T13:30:28.686 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-01T13:30:28.686 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-01T13:30:28.686 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_journal_size": "5120", 2024-04-01T13:30:28.687 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_kill_backfill_at": "0", 2024-04-01T13:30:28.687 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-01T13:30:28.687 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_map_cache_size": "50", 2024-04-01T13:30:28.687 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_map_dedup": "true", 2024-04-01T13:30:28.687 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_map_message_max": "40", 2024-04-01T13:30:28.687 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-01T13:30:28.687 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_map_share_max_epochs": "40", 2024-04-01T13:30:28.687 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_attr_name_len": "100", 2024-04-01T13:30:28.687 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_attr_size": "0", 2024-04-01T13:30:28.687 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_backfills": "1", 2024-04-01T13:30:28.687 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_markdown_count": "5", 2024-04-01T13:30:28.687 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_markdown_period": "600", 2024-04-01T13:30:28.687 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_object_name_len": "2048", 2024-04-01T13:30:28.687 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_object_namespace_len": "256", 2024-04-01T13:30:28.687 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_object_size": "134217728", 2024-04-01T13:30:28.688 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-01T13:30:28.688 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-01T13:30:28.688 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_pg_blocked_by": "16", 2024-04-01T13:30:28.688 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_pg_log_entries": "10000", 2024-04-01T13:30:28.688 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-01T13:30:28.688 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_pgls": "1024", 2024-04-01T13:30:28.688 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_push_cost": "8388608", 2024-04-01T13:30:28.688 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_push_objects": "10", 2024-04-01T13:30:28.688 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_scrubs": "3", 2024-04-01T13:30:28.688 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-01T13:30:28.688 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_trimming_pgs": "2", 2024-04-01T13:30:28.688 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_write_op_reply_len": "64", 2024-04-01T13:30:28.688 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_write_size": "90", 2024-04-01T13:30:28.688 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-01T13:30:28.688 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-01T13:30:28.689 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-01T13:30:28.689 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-01T13:30:28.689 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-01T13:30:28.689 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-01T13:30:28.689 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-01T13:30:28.689 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-01T13:30:28.689 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_profile": "balanced", 2024-04-01T13:30:28.689 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-01T13:30:28.689 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-01T13:30:28.689 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-01T13:30:28.689 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-01T13:30:28.689 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-01T13:30:28.689 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-01T13:30:28.689 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-01T13:30:28.689 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-01T13:30:28.690 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-01T13:30:28.690 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-01T13:30:28.690 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-01T13:30:28.690 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_base": "805306368", 2024-04-01T13:30:28.690 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_cache_min": "134217728", 2024-04-01T13:30:28.690 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-01T13:30:28.690 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-01T13:30:28.690 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_target": "4294967296", 2024-04-01T13:30:28.690 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_target_autotune": "false", 2024-04-01T13:30:28.690 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-01T13:30:28.690 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_min_pg_log_entries": "250", 2024-04-01T13:30:28.690 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_min_recovery_priority": "0", 2024-04-01T13:30:28.690 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-01T13:30:28.690 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-01T13:30:28.690 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mon_report_interval": "5", 2024-04-01T13:30:28.691 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-01T13:30:28.691 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-01T13:30:28.691 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_num_cache_shards": "32", 2024-04-01T13:30:28.691 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_num_op_tracker_shard": "32", 2024-04-01T13:30:28.691 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_numa_auto_affinity": "true", 2024-04-01T13:30:28.691 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_numa_node": "-1", 2024-04-01T13:30:28.691 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_numa_prefer_iface": "true", 2024-04-01T13:30:28.691 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-01T13:30:28.691 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_objecter_finishers": "1", 2024-04-01T13:30:28.691 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_objectstore": "bluestore", 2024-04-01T13:30:28.691 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_objectstore_fuse": "false", 2024-04-01T13:30:28.691 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_objectstore_tracing": "false", 2024-04-01T13:30:28.691 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_complaint_time": "30.000000", 2024-04-01T13:30:28.691 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_history_duration": "600", 2024-04-01T13:30:28.691 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_history_size": "20", 2024-04-01T13:30:28.692 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_history_slow_op_size": "20", 2024-04-01T13:30:28.692 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-01T13:30:28.692 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_log_threshold": "5", 2024-04-01T13:30:28.692 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_num_shards": "0", 2024-04-01T13:30:28.692 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_num_shards_hdd": "5", 2024-04-01T13:30:28.692 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_num_shards_ssd": "8", 2024-04-01T13:30:28.692 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-01T13:30:28.692 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-01T13:30:28.692 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-01T13:30:28.692 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-01T13:30:28.692 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_pq_min_cost": "65536", 2024-04-01T13:30:28.692 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-01T13:30:28.692 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_queue_cut_off": "high", 2024-04-01T13:30:28.692 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-01T13:30:28.692 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_thread_timeout": "15", 2024-04-01T13:30:28.692 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_open_classes_on_start": "true", 2024-04-01T13:30:28.693 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_os_flags": "0", 2024-04-01T13:30:28.693 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_peering_op_priority": "255", 2024-04-01T13:30:28.693 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_delete_cost": "1048576", 2024-04-01T13:30:28.693 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_delete_priority": "5", 2024-04-01T13:30:28.693 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-01T13:30:28.693 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-01T13:30:28.693 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-01T13:30:28.693 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_log_trim_max": "10000", 2024-04-01T13:30:28.693 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_log_trim_min": "100", 2024-04-01T13:30:28.693 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-01T13:30:28.693 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-01T13:30:28.693 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-01T13:30:28.693 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-01T13:30:28.693 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-01T13:30:28.693 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-01T13:30:28.694 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-01T13:30:28.694 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-01T13:30:28.694 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-01T13:30:28.694 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-01T13:30:28.694 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_crimson": "true", 2024-04-01T13:30:28.694 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-01T13:30:28.694 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-01T13:30:28.694 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-01T13:30:28.694 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-01T13:30:28.694 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-01T13:30:28.694 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-01T13:30:28.694 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-01T13:30:28.694 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-01T13:30:28.694 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_flags": "0", 2024-04-01T13:30:28.694 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-01T13:30:28.695 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_min_size": "2", 2024-04-01T13:30:28.695 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-01T13:30:28.695 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_pg_num": "32", 2024-04-01T13:30:28.695 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_pgp_num": "0", 2024-04-01T13:30:28.695 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-01T13:30:28.695 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_read_ratio": "70", 2024-04-01T13:30:28.695 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_size": "3", 2024-04-01T13:30:28.695 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_type": "replicated", 2024-04-01T13:30:28.695 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-01T13:30:28.695 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-01T13:30:28.695 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_push_per_object_cost": "1000", 2024-04-01T13:30:28.695 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-01T13:30:28.695 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recover_clone_overlap": "true", 2024-04-01T13:30:28.695 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-01T13:30:28.696 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_cost": "20971520", 2024-04-01T13:30:28.696 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-01T13:30:28.696 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_max_active": "0", 2024-04-01T13:30:28.696 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-01T13:30:28.696 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-01T13:30:28.696 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-01T13:30:28.696 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-01T13:30:28.696 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_max_single_start": "1", 2024-04-01T13:30:28.696 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_op_priority": "3", 2024-04-01T13:30:28.696 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-01T13:30:28.696 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_priority": "5", 2024-04-01T13:30:28.696 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-01T13:30:28.696 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_sleep": "0.000000", 2024-04-01T13:30:28.696 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-01T13:30:28.696 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-01T13:30:28.697 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-01T13:30:28.697 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_repair_during_recovery": "false", 2024-04-01T13:30:28.697 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_requested_scrub_priority": "120", 2024-04-01T13:30:28.697 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-01T13:30:28.697 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-01T13:30:28.697 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_auto_repair": "false", 2024-04-01T13:30:28.697 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-01T13:30:28.697 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-01T13:30:28.697 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_begin_hour": "0", 2024-04-01T13:30:28.697 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_begin_week_day": "0", 2024-04-01T13:30:28.697 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_chunk_max": "25", 2024-04-01T13:30:28.697 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_chunk_min": "5", 2024-04-01T13:30:28.697 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_cost": "52428800", 2024-04-01T13:30:28.698 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-01T13:30:28.698 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_during_recovery": "false", 2024-04-01T13:30:28.698 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_end_hour": "0", 2024-04-01T13:30:28.698 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_end_week_day": "0", 2024-04-01T13:30:28.698 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_event_cost": "4096", 2024-04-01T13:30:28.698 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-01T13:30:28.698 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-01T13:30:28.698 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_invalid_stats": "true", 2024-04-01T13:30:28.698 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-01T13:30:28.698 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-01T13:30:28.698 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_max_preemptions": "5", 2024-04-01T13:30:28.698 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-01T13:30:28.699 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_priority": "5", 2024-04-01T13:30:28.699 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-01T13:30:28.699 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_sleep": "0.000000", 2024-04-01T13:30:28.699 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-01T13:30:28.699 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-01T13:30:28.699 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-01T13:30:28.699 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-04-01T13:30:28.699 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_skip_data_digest": "false", 2024-04-01T13:30:28.699 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_smart_report_timeout": "5", 2024-04-01T13:30:28.699 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_snap_trim_cost": "1048576", 2024-04-01T13:30:28.700 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_snap_trim_priority": "5", 2024-04-01T13:30:28.700 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-01T13:30:28.700 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-01T13:30:28.700 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-01T13:30:28.700 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-01T13:30:28.700 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-01T13:30:28.700 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-01T13:30:28.700 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-01T13:30:28.700 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_target_transaction_size": "30", 2024-04-01T13:30:28.700 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-01T13:30:28.700 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-01T13:30:28.700 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-01T13:30:28.700 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-01T13:30:28.700 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-01T13:30:28.700 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-01T13:30:28.701 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-01T13:30:28.701 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-01T13:30:28.701 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-01T13:30:28.701 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-01T13:30:28.701 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tracing": "false", 2024-04-01T13:30:28.701 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_use_stale_snap": "false", 2024-04-01T13:30:28.701 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-01T13:30:28.701 INFO:teuthology.orchestra.run.smithi059.stdout: "osdc_blkin_trace_all": "false", 2024-04-01T13:30:28.701 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_kill_at": "0", 2024-04-01T13:30:28.701 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_max_join_drift": "10", 2024-04-01T13:30:28.701 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_min": "500", 2024-04-01T13:30:28.701 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_min_wait": "0.050000", 2024-04-01T13:30:28.701 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_propose_interval": "1.000000", 2024-04-01T13:30:28.701 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_service_trim_max": "500", 2024-04-01T13:30:28.701 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-01T13:30:28.702 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_service_trim_min": "10", 2024-04-01T13:30:28.702 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_stash_full_interval": "25", 2024-04-01T13:30:28.702 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_trim_max": "500", 2024-04-01T13:30:28.702 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_trim_min": "250", 2024-04-01T13:30:28.702 INFO:teuthology.orchestra.run.smithi059.stdout: "perf": "true", 2024-04-01T13:30:28.702 INFO:teuthology.orchestra.run.smithi059.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-01T13:30:28.702 INFO:teuthology.orchestra.run.smithi059.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-01T13:30:28.702 INFO:teuthology.orchestra.run.smithi059.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-01T13:30:28.702 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "-", 2024-04-01T13:30:28.702 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrv": "", 2024-04-01T13:30:28.702 INFO:teuthology.orchestra.run.smithi059.stdout: "public_bind_addr": "-", 2024-04-01T13:30:28.702 INFO:teuthology.orchestra.run.smithi059.stdout: "public_network": "", 2024-04-01T13:30:28.702 INFO:teuthology.orchestra.run.smithi059.stdout: "public_network_interface": "", 2024-04-01T13:30:28.702 INFO:teuthology.orchestra.run.smithi059.stdout: "qat_compressor_enabled": "false", 2024-04-01T13:30:28.702 INFO:teuthology.orchestra.run.smithi059.stdout: "qat_compressor_session_max_number": "256", 2024-04-01T13:30:28.703 INFO:teuthology.orchestra.run.smithi059.stdout: "rados_mon_op_timeout": "0", 2024-04-01T13:30:28.703 INFO:teuthology.orchestra.run.smithi059.stdout: "rados_osd_op_timeout": "0", 2024-04-01T13:30:28.703 INFO:teuthology.orchestra.run.smithi059.stdout: "rados_tracing": "false", 2024-04-01T13:30:28.703 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_atime_update_interval": "60", 2024-04-01T13:30:28.703 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-01T13:30:28.703 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_balance_parent_reads": "false", 2024-04-01T13:30:28.703 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_balance_snap_reads": "false", 2024-04-01T13:30:28.703 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_blkin_trace_all": "false", 2024-04-01T13:30:28.703 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-01T13:30:28.703 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-01T13:30:28.703 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache": "true", 2024-04-01T13:30:28.703 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-01T13:30:28.703 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-01T13:30:28.703 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-01T13:30:28.703 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-01T13:30:28.704 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_policy": "writearound", 2024-04-01T13:30:28.704 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_size": "33554432", 2024-04-01T13:30:28.704 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-01T13:30:28.704 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-01T13:30:28.704 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_clone_copy_on_read": "false", 2024-04-01T13:30:28.704 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_compression_hint": "none", 2024-04-01T13:30:28.704 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_concurrent_management_ops": "10", 2024-04-01T13:30:28.704 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-01T13:30:28.704 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_clone_format": "auto", 2024-04-01T13:30:28.704 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_data_pool": "", 2024-04-01T13:30:28.704 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_features": "61", 2024-04-01T13:30:28.704 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_format": "2", 2024-04-01T13:30:28.704 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_map_options": "", 2024-04-01T13:30:28.704 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_order": "22", 2024-04-01T13:30:28.704 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_pool": "rbd", 2024-04-01T13:30:28.704 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-01T13:30:28.705 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_stripe_count": "0", 2024-04-01T13:30:28.705 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_stripe_unit": "0", 2024-04-01T13:30:28.705 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-01T13:30:28.705 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-01T13:30:28.705 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-01T13:30:28.705 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_enable_alloc_hint": "true", 2024-04-01T13:30:28.705 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-01T13:30:28.705 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_io_scheduler": "simple", 2024-04-01T13:30:28.705 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-01T13:30:28.705 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-01T13:30:28.705 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-01T13:30:28.705 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-01T13:30:28.705 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-01T13:30:28.705 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-01T13:30:28.705 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-01T13:30:28.706 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-01T13:30:28.706 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-01T13:30:28.706 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_order": "24", 2024-04-01T13:30:28.706 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_pool": "", 2024-04-01T13:30:28.706 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_splay_width": "4", 2024-04-01T13:30:28.706 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_localize_parent_reads": "false", 2024-04-01T13:30:28.706 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_localize_snap_reads": "false", 2024-04-01T13:30:28.706 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-01T13:30:28.706 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-01T13:30:28.706 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-01T13:30:28.706 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-01T13:30:28.706 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-01T13:30:28.706 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-01T13:30:28.706 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-01T13:30:28.707 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-01T13:30:28.707 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-01T13:30:28.707 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-01T13:30:28.707 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-01T13:30:28.707 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-01T13:30:28.707 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-01T13:30:28.707 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-01T13:30:28.707 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-01T13:30:28.707 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-01T13:30:28.707 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-01T13:30:28.707 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-01T13:30:28.707 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-01T13:30:28.707 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-01T13:30:28.707 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-01T13:30:28.707 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-01T13:30:28.707 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-01T13:30:28.708 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-01T13:30:28.708 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-01T13:30:28.708 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-01T13:30:28.708 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-01T13:30:28.708 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-01T13:30:28.708 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-01T13:30:28.708 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-01T13:30:28.708 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-01T13:30:28.708 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mtime_update_interval": "60", 2024-04-01T13:30:28.708 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_non_blocking_aio": "true", 2024-04-01T13:30:28.708 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_op_thread_timeout": "60", 2024-04-01T13:30:28.708 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_op_threads": "1", 2024-04-01T13:30:28.708 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_parent_cache_enabled": "false", 2024-04-01T13:30:28.708 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-01T13:30:28.709 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-01T13:30:28.709 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-01T13:30:28.709 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_plugins": "", 2024-04-01T13:30:28.709 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_bps_burst": "0", 2024-04-01T13:30:28.709 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-01T13:30:28.709 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_bps_limit": "0", 2024-04-01T13:30:28.709 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_exclude_ops": "0", 2024-04-01T13:30:28.709 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_iops_burst": "0", 2024-04-01T13:30:28.709 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-01T13:30:28.709 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_iops_limit": "0", 2024-04-01T13:30:28.709 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-01T13:30:28.709 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-01T13:30:28.709 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-01T13:30:28.709 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-01T13:30:28.709 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-01T13:30:28.710 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-01T13:30:28.710 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-01T13:30:28.710 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-01T13:30:28.710 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-01T13:30:28.710 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-01T13:30:28.710 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-01T13:30:28.710 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-01T13:30:28.710 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-01T13:30:28.710 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-01T13:30:28.710 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_read_from_replica_policy": "default", 2024-04-01T13:30:28.710 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-01T13:30:28.710 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-01T13:30:28.710 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-01T13:30:28.710 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-01T13:30:28.711 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_skip_partial_discard": "true", 2024-04-01T13:30:28.711 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-01T13:30:28.711 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_tracing": "false", 2024-04-01T13:30:28.711 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_validate_names": "true", 2024-04-01T13:30:28.711 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_validate_pool": "true", 2024-04-01T13:30:28.711 INFO:teuthology.orchestra.run.smithi059.stdout: "restapi_base_url": "", 2024-04-01T13:30:28.711 INFO:teuthology.orchestra.run.smithi059.stdout: "restapi_log_level": "", 2024-04-01T13:30:28.711 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_acl_grants_max_num": "100", 2024-04-01T13:30:28.711 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_admin_entry": "admin", 2024-04-01T13:30:28.711 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-01T13:30:28.711 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_backend_store": "rados", 2024-04-01T13:30:28.711 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_barbican_url": "", 2024-04-01T13:30:28.711 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_beast_enable_async": "true", 2024-04-01T13:30:28.711 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_counters_cache": "false", 2024-04-01T13:30:28.711 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-01T13:30:28.712 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-01T13:30:28.712 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-01T13:30:28.712 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-01T13:30:28.712 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-01T13:30:28.712 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-01T13:30:28.712 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-01T13:30:28.712 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-01T13:30:28.712 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_cache_enabled": "true", 2024-04-01T13:30:28.712 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_cache_expiry_interval": "900", 2024-04-01T13:30:28.712 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_cache_lru_size": "10000", 2024-04-01T13:30:28.712 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_config_store": "rados", 2024-04-01T13:30:28.712 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_content_length_compat": "false", 2024-04-01T13:30:28.712 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_copy_obj_progress": "true", 2024-04-01T13:30:28.712 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-01T13:30:28.713 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_cors_rules_max_num": "100", 2024-04-01T13:30:28.713 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_cross_domain_policy": "", 2024-04-01T13:30:28.713 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-01T13:30:28.713 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_addr": "", 2024-04-01T13:30:28.713 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-01T13:30:28.713 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-01T13:30:28.713 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-01T13:30:28.713 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-01T13:30:28.713 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_password": "", 2024-04-01T13:30:28.713 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-01T13:30:28.713 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_username": "", 2024-04-01T13:30:28.713 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_require_ssl": "true", 2024-04-01T13:30:28.713 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-01T13:30:28.713 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-01T13:30:28.713 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-01T13:30:28.714 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-01T13:30:28.714 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-01T13:30:28.714 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-01T13:30:28.714 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-01T13:30:28.714 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-01T13:30:28.714 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-01T13:30:28.714 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-01T13:30:28.714 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-01T13:30:28.714 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-01T13:30:28.714 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-01T13:30:28.714 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-01T13:30:28.714 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-01T13:30:28.714 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_addr": "", 2024-04-01T13:30:28.714 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_auth": "token", 2024-04-01T13:30:28.714 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_namespace": "", 2024-04-01T13:30:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_prefix": "", 2024-04-01T13:30:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-01T13:30:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-01T13:30:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-01T13:30:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-01T13:30:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_token_file": "", 2024-04-01T13:30:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-01T13:30:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_curl_buffersize": "524288", 2024-04-01T13:30:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-01T13:30:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_curl_low_speed_time": "300", 2024-04-01T13:30:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-01T13:30:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-01T13:30:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-01T13:30:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-01T13:30:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-01T13:30:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-01T13:30:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-01T13:30:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-01T13:30:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-01T13:30:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-01T13:30:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-01T13:30:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d4n_port": "6379", 2024-04-01T13:30:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-01T13:30:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_log_changes_size": "1000", 2024-04-01T13:30:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_log_num_shards": "128", 2024-04-01T13:30:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-01T13:30:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_log_window": "30", 2024-04-01T13:30:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-01T13:30:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-01T13:30:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-01T13:30:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-01T13:30:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-01T13:30:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-01T13:30:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-01T13:30:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-01T13:30:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-01T13:30:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-01T13:30:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-01T13:30:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-01T13:30:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-01T13:30:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-01T13:30:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-01T13:30:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-01T13:30:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-01T13:30:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-01T13:30:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-01T13:30:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-01T13:30:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-01T13:30:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-01T13:30:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-01T13:30:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-01T13:30:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-01T13:30:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dns_name": "", 2024-04-01T13:30:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dns_s3website_name": "", 2024-04-01T13:30:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dynamic_resharding": "true", 2024-04-01T13:30:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-01T13:30:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_gc_threads": "true", 2024-04-01T13:30:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_lc_threads": "true", 2024-04-01T13:30:28.719 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_ops_log": "false", 2024-04-01T13:30:28.719 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_quota_threads": "true", 2024-04-01T13:30:28.719 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_static_website": "false", 2024-04-01T13:30:28.719 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_usage_log": "false", 2024-04-01T13:30:28.719 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enforce_swift_acls": "true", 2024-04-01T13:30:28.719 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_exit_timeout_secs": "120", 2024-04-01T13:30:28.719 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_expose_bucket": "false", 2024-04-01T13:30:28.719 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_extended_http_attrs": "", 2024-04-01T13:30:28.719 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_filter": "none", 2024-04-01T13:30:28.719 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-01T13:30:28.719 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_frontends": "beast port=7480", 2024-04-01T13:30:28.719 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-01T13:30:28.719 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_max_deferred": "50", 2024-04-01T13:30:28.719 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-01T13:30:28.719 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_max_objs": "32", 2024-04-01T13:30:28.720 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-01T13:30:28.720 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-01T13:30:28.720 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-01T13:30:28.720 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-01T13:30:28.720 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_processor_period": "3600", 2024-04-01T13:30:28.720 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-01T13:30:28.720 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-01T13:30:28.720 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-01T13:30:28.720 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-01T13:30:28.720 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_init_timeout": "300", 2024-04-01T13:30:28.720 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-01T13:30:28.720 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-01T13:30:28.720 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_kafka_connection_idle": "300", 2024-04-01T13:30:28.720 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-01T13:30:28.720 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-01T13:30:28.721 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-01T13:30:28.721 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-01T13:30:28.721 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-01T13:30:28.721 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_domain": "", 2024-04-01T13:30:28.721 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_password": "", 2024-04-01T13:30:28.721 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_password_path": "", 2024-04-01T13:30:28.721 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_project": "", 2024-04-01T13:30:28.721 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_tenant": "", 2024-04-01T13:30:28.721 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_token": "", 2024-04-01T13:30:28.721 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_token_path": "", 2024-04-01T13:30:28.721 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_user": "", 2024-04-01T13:30:28.721 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_api_version": "2", 2024-04-01T13:30:28.721 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_barbican_domain": "", 2024-04-01T13:30:28.721 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_barbican_password": "", 2024-04-01T13:30:28.721 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_barbican_project": "", 2024-04-01T13:30:28.722 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-01T13:30:28.722 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_barbican_user": "", 2024-04-01T13:30:28.722 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-01T13:30:28.722 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-01T13:30:28.722 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-01T13:30:28.722 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-01T13:30:28.722 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-01T13:30:28.722 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_url": "", 2024-04-01T13:30:28.722 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-01T13:30:28.722 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_debug_interval": "-1", 2024-04-01T13:30:28.722 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_lock_max_time": "90", 2024-04-01T13:30:28.722 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_max_objs": "32", 2024-04-01T13:30:28.722 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_max_rules": "1000", 2024-04-01T13:30:28.722 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_max_worker": "3", 2024-04-01T13:30:28.722 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-01T13:30:28.723 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_thread_delay": "0", 2024-04-01T13:30:28.723 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-01T13:30:28.723 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ldap_dnattr": "uid", 2024-04-01T13:30:28.723 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-01T13:30:28.723 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ldap_searchfilter": "", 2024-04-01T13:30:28.723 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-01T13:30:28.723 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ldap_uri": "ldaps://", 2024-04-01T13:30:28.723 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-01T13:30:28.723 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-01T13:30:28.723 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-01T13:30:28.723 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_log_http_headers": "", 2024-04-01T13:30:28.723 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-01T13:30:28.723 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-01T13:30:28.723 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_log_object_name_utc": "false", 2024-04-01T13:30:28.723 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-01T13:30:28.724 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-01T13:30:28.724 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_attr_name_len": "0", 2024-04-01T13:30:28.724 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_attr_size": "0", 2024-04-01T13:30:28.724 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-01T13:30:28.724 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_chunk_size": "4194304", 2024-04-01T13:30:28.724 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-01T13:30:28.724 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-01T13:30:28.724 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-01T13:30:28.724 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_listing_results": "1000", 2024-04-01T13:30:28.724 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_notify_retries": "10", 2024-04-01T13:30:28.724 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-01T13:30:28.724 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_put_param_size": "1048576", 2024-04-01T13:30:28.724 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_put_size": "5368709120", 2024-04-01T13:30:28.724 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_slo_entries": "1000", 2024-04-01T13:30:28.724 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_md_log_max_shards": "64", 2024-04-01T13:30:28.725 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-01T13:30:28.725 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-01T13:30:28.725 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-01T13:30:28.725 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-01T13:30:28.725 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_mp_lock_max_time": "600", 2024-04-01T13:30:28.725 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-01T13:30:28.725 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-01T13:30:28.725 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-01T13:30:28.725 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-01T13:30:28.725 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-01T13:30:28.725 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-01T13:30:28.725 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-01T13:30:28.725 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-01T13:30:28.725 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_max_gc": "300", 2024-04-01T13:30:28.725 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-01T13:30:28.726 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-01T13:30:28.726 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-01T13:30:28.726 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-01T13:30:28.726 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-01T13:30:28.726 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-01T13:30:28.726 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-01T13:30:28.726 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_num_async_rados_threads": "32", 2024-04-01T13:30:28.726 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_num_control_oids": "8", 2024-04-01T13:30:28.726 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_numa_node": "-1", 2024-04-01T13:30:28.726 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-01T13:30:28.726 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-01T13:30:28.726 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_objexp_chunk_size": "100", 2024-04-01T13:30:28.726 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_objexp_gc_interval": "600", 2024-04-01T13:30:28.726 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-01T13:30:28.727 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-01T13:30:28.727 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-01T13:30:28.727 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_op_thread_timeout": "600", 2024-04-01T13:30:28.727 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_op_tracing": "false", 2024-04-01T13:30:28.727 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_opa_token": "", 2024-04-01T13:30:28.727 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_opa_url": "", 2024-04-01T13:30:28.727 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_opa_verify_ssl": "true", 2024-04-01T13:30:28.727 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-01T13:30:28.727 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-01T13:30:28.727 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ops_log_rados": "false", 2024-04-01T13:30:28.727 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ops_log_socket_path": "", 2024-04-01T13:30:28.727 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-01T13:30:28.727 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-01T13:30:28.727 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-01T13:30:28.727 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_period_push_interval": "2.000000", 2024-04-01T13:30:28.728 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-01T13:30:28.728 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-01T13:30:28.728 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-01T13:30:28.728 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-01T13:30:28.728 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_posix_cache_lanes": "3", 2024-04-01T13:30:28.728 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-01T13:30:28.728 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-01T13:30:28.728 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_posix_cache_partitions": "3", 2024-04-01T13:30:28.728 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-01T13:30:28.728 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_print_continue": "true", 2024-04-01T13:30:28.728 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-01T13:30:28.728 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-01T13:30:28.728 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-01T13:30:28.728 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-01T13:30:28.728 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-01T13:30:28.729 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_rados_tracing": "false", 2024-04-01T13:30:28.729 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_realm": "", 2024-04-01T13:30:28.729 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_realm_id": "", 2024-04-01T13:30:28.729 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-01T13:30:28.729 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_region": "", 2024-04-01T13:30:28.729 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-01T13:30:28.729 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-01T13:30:28.729 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-01T13:30:28.729 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-01T13:30:28.729 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_request_uri": "", 2024-04-01T13:30:28.729 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_reshard_batch_size": "64", 2024-04-01T13:30:28.729 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-01T13:30:28.729 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_reshard_max_aio": "128", 2024-04-01T13:30:28.729 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_reshard_num_logs": "16", 2024-04-01T13:30:28.729 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_reshard_thread_interval": "600", 2024-04-01T13:30:28.730 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_resolve_cname": "false", 2024-04-01T13:30:28.730 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-01T13:30:28.730 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_run_sync_thread": "true", 2024-04-01T13:30:28.730 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-01T13:30:28.730 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-01T13:30:28.730 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-01T13:30:28.730 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-01T13:30:28.730 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-01T13:30:28.730 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-01T13:30:28.730 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-01T13:30:28.730 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-01T13:30:28.730 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-01T13:30:28.730 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_scheduler_type": "throttler", 2024-04-01T13:30:28.730 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_script_uri": "", 2024-04-01T13:30:28.730 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_service_provider_name": "", 2024-04-01T13:30:28.731 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-01T13:30:28.731 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_client_id": "", 2024-04-01T13:30:28.731 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_client_secret": "", 2024-04-01T13:30:28.731 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_entry": "sts", 2024-04-01T13:30:28.731 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_key": "sts", 2024-04-01T13:30:28.731 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-01T13:30:28.731 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_min_session_duration": "900", 2024-04-01T13:30:28.731 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_token_introspection_url": "", 2024-04-01T13:30:28.731 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_account_in_url": "false", 2024-04-01T13:30:28.731 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_auth_entry": "auth", 2024-04-01T13:30:28.731 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_auth_url": "", 2024-04-01T13:30:28.731 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_custom_header": "", 2024-04-01T13:30:28.731 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-01T13:30:28.731 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_need_stats": "true", 2024-04-01T13:30:28.731 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_tenant_name": "", 2024-04-01T13:30:28.732 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_token_expiration": "86400", 2024-04-01T13:30:28.732 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_url": "", 2024-04-01T13:30:28.732 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_url_prefix": "swift", 2024-04-01T13:30:28.732 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-01T13:30:28.732 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-01T13:30:28.732 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-01T13:30:28.732 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_lease_period": "120", 2024-04-01T13:30:28.732 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-01T13:30:28.732 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-01T13:30:28.732 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-01T13:30:28.732 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-01T13:30:28.732 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-01T13:30:28.732 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-01T13:30:28.732 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-01T13:30:28.732 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-01T13:30:28.733 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-01T13:30:28.733 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_thread_pool_size": "512", 2024-04-01T13:30:28.733 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-01T13:30:28.733 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-01T13:30:28.733 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-01T13:30:28.733 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-01T13:30:28.733 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_comment": "", 2024-04-01T13:30:28.733 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_createby": "", 2024-04-01T13:30:28.733 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_encoding": "", 2024-04-01T13:30:28.733 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_flag": "false", 2024-04-01T13:30:28.733 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-01T13:30:28.733 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_origin": "", 2024-04-01T13:30:28.733 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-01T13:30:28.733 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_tracker": "", 2024-04-01T13:30:28.733 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_trust_forwarded_https": "false", 2024-04-01T13:30:28.733 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-01T13:30:28.734 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-01T13:30:28.734 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_usage_max_shards": "32", 2024-04-01T13:30:28.734 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_usage_max_user_shards": "1", 2024-04-01T13:30:28.734 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_use_opa_authz": "false", 2024-04-01T13:30:28.734 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_counters_cache": "false", 2024-04-01T13:30:28.734 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-01T13:30:28.734 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-01T13:30:28.734 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-01T13:30:28.734 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_max_buckets": "1000", 2024-04-01T13:30:28.734 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_policies_max_num": "100", 2024-04-01T13:30:28.734 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-01T13:30:28.734 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-01T13:30:28.734 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-01T13:30:28.734 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-01T13:30:28.734 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_unique_email": "true", 2024-04-01T13:30:28.734 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_verify_ssl": "true", 2024-04-01T13:30:28.735 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-01T13:30:28.735 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_zone": "", 2024-04-01T13:30:28.735 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_zone_id": "", 2024-04-01T13:30:28.735 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-01T13:30:28.735 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_zonegroup": "", 2024-04-01T13:30:28.735 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_zonegroup_id": "", 2024-04-01T13:30:28.735 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-01T13:30:28.735 INFO:teuthology.orchestra.run.smithi059.stdout: "rgwlc_auto_session_clear": "true", 2024-04-01T13:30:28.735 INFO:teuthology.orchestra.run.smithi059.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-01T13:30:28.735 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_block_size": "4096", 2024-04-01T13:30:28.735 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-01T13:30:28.735 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-01T13:30:28.735 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-01T13:30:28.735 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-01T13:30:28.735 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-01T13:30:28.736 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cache_size": "536870912", 2024-04-01T13:30:28.736 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-01T13:30:28.736 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-01T13:30:28.736 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-01T13:30:28.736 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-01T13:30:28.736 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-01T13:30:28.736 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-01T13:30:28.736 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-01T13:30:28.736 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-01T13:30:28.736 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_index_type": "binary_search", 2024-04-01T13:30:28.736 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-01T13:30:28.736 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-01T13:30:28.736 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_partition_filters": "false", 2024-04-01T13:30:28.736 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_perf": "false", 2024-04-01T13:30:28.736 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-01T13:30:28.736 INFO:teuthology.orchestra.run.smithi059.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-01T13:30:28.736 INFO:teuthology.orchestra.run.smithi059.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-01T13:30:28.737 INFO:teuthology.orchestra.run.smithi059.stdout: "run_dir": "/var/run/ceph", 2024-04-01T13:30:28.737 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_block_create": "true", 2024-04-01T13:30:28.737 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_cache_lru_size": "67108864", 2024-04-01T13:30:28.737 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-01T13:30:28.737 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-01T13:30:28.737 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_default_max_object_size": "16777216", 2024-04-01T13:30:28.737 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-01T13:30:28.737 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_device_size": "53687091200", 2024-04-01T13:30:28.737 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_journal_batch_capacity": "16", 2024-04-01T13:30:28.737 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-01T13:30:28.737 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-01T13:30:28.737 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-01T13:30:28.737 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_main_device_type": "SSD", 2024-04-01T13:30:28.737 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-01T13:30:28.737 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-01T13:30:28.738 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-01T13:30:28.738 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-01T13:30:28.738 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-01T13:30:28.738 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_segment_size": "67108864", 2024-04-01T13:30:28.738 INFO:teuthology.orchestra.run.smithi059.stdout: "set_keepcaps": "false", 2024-04-01T13:30:28.738 INFO:teuthology.orchestra.run.smithi059.stdout: "setgroup": "", 2024-04-01T13:30:28.738 INFO:teuthology.orchestra.run.smithi059.stdout: "setuser": "", 2024-04-01T13:30:28.738 INFO:teuthology.orchestra.run.smithi059.stdout: "setuser_match_path": "", 2024-04-01T13:30:28.738 INFO:teuthology.orchestra.run.smithi059.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-01T13:30:28.738 INFO:teuthology.orchestra.run.smithi059.stdout: "thp": "false", 2024-04-01T13:30:28.738 INFO:teuthology.orchestra.run.smithi059.stdout: "threadpool_default_timeout": "60", 2024-04-01T13:30:28.738 INFO:teuthology.orchestra.run.smithi059.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-01T13:30:28.739 INFO:teuthology.orchestra.run.smithi059.stdout: "throttler_perf_counter": "true" 2024-04-01T13:30:28.739 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-01T13:30:28.741 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-01T13:30:28.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:28.862+0000 7f69a286d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:28.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:28.868+0000 7f69a286d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:28.870 INFO:teuthology.orchestra.run.smithi059.stdout:{} 2024-04-01T13:30:28.871 INFO:teuthology.orchestra.run.smithi059.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-01T13:30:28.881 DEBUG:teuthology.orchestra.run.smithi087:> type ceph-objectstore-tool 2024-04-01T13:30:28.899 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-01T13:30:28.899 DEBUG:teuthology.orchestra.run.smithi059:> type ceph-objectstore-tool 2024-04-01T13:30:28.912 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-01T13:30:28.913 INFO:teuthology.run_tasks:Running task rados... 2024-04-01T13:30:28.924 INFO:tasks.rados:Beginning rados... 2024-04-01T13:30:28.924 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000, 'pool_snaps': True} 2024-04-01T13:30:28.924 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-matanb-crimson-testing-thrash-options'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'afa73a9e21b8fb0610f8af7266f1975be37c0a69'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'conf': {'osd': {'osd deep scrub update digest min age': 0}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'afa73a9e21b8fb0610f8af7266f1975be37c0a69'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-matanb-crimson-testing-thrash-options', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'afa73a9e21b8fb0610f8af7266f1975be37c0a69'}} 2024-04-01T13:30:28.924 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000, 'pool_snaps': True} 2024-04-01T13:30:28.924 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-04-01T13:30:28.971 INFO:tasks.rados:joining rados 2024-04-01T13:30:28.971 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-01T13:30:28.971 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-01T13:30:28.971 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-01T13:30:28.971 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T13:30:28.972 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-01T13:30:28.972 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T13:30:28.972 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2024-04-01T13:30:28.972 INFO:tasks.rados:starting run 0 out of 1 2024-04-01T13:30:28.972 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-01T13:30:28.972 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-04-01T13:30:29.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:29.100+0000 7f9a02554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:29.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:29.100+0000 7f55adc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:29.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:29.107+0000 7f9a02554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:29.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:29.100+0000 7f9c6d15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:29.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:29.109+0000 7f55adc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:29.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:29.114+0000 7f9c6d15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:29.303 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:30:29.303 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-01T13:30:29.315 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-01T13:30:27.365356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107912,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379448,"statfs":{"total":386547056640,"available":386436554752,"internally_reserved":0,"allocated":593920,"data_stored":1870024,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.965052"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T13:30:25.407393+0000","last_change":"2024-04-01T13:30:24.495962+0000","last_active":"2024-04-01T13:30:25.407393+0000","last_peered":"2024-04-01T13:30:25.407393+0000","last_clean":"2024-04-01T13:30:25.407393+0000","last_became_active":"2024-04-01T13:30:24.489422+0000","last_became_peered":"2024-04-01T13:30:24.489422+0000","last_unstale":"2024-04-01T13:30:25.407393+0000","last_undegraded":"2024-04-01T13:30:25.407393+0000","last_fullsized":"2024-04-01T13:30:25.407393+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.525265+0000","last_change":"2024-04-01T13:30:24.526346+0000","last_active":"2024-04-01T13:30:24.525265+0000","last_peered":"2024-04-01T13:30:24.525265+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.521930+0000","last_became_peered":"2024-04-01T13:30:24.521930+0000","last_unstale":"2024-04-01T13:30:24.525265+0000","last_undegraded":"2024-04-01T13:30:24.525265+0000","last_fullsized":"2024-04-01T13:30:24.525265+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.480277+0000","last_change":"2024-04-01T13:30:24.481500+0000","last_active":"2024-04-01T13:30:24.480277+0000","last_peered":"2024-04-01T13:30:24.480277+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.476810+0000","last_became_peered":"2024-04-01T13:30:24.476810+0000","last_unstale":"2024-04-01T13:30:24.480277+0000","last_undegraded":"2024-04-01T13:30:24.480277+0000","last_fullsized":"2024-04-01T13:30:24.480277+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.498967+0000","last_change":"2024-04-01T13:30:24.499851+0000","last_active":"2024-04-01T13:30:24.498967+0000","last_peered":"2024-04-01T13:30:24.498967+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.495368+0000","last_became_peered":"2024-04-01T13:30:24.495368+0000","last_unstale":"2024-04-01T13:30:24.498967+0000","last_undegraded":"2024-04-01T13:30:24.498967+0000","last_fullsized":"2024-04-01T13:30:24.498967+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.478572+0000","last_change":"2024-04-01T13:30:24.480593+0000","last_active":"2024-04-01T13:30:24.478572+0000","last_peered":"2024-04-01T13:30:24.478572+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.472751+0000","last_became_peered":"2024-04-01T13:30:24.472751+0000","last_unstale":"2024-04-01T13:30:24.478572+0000","last_undegraded":"2024-04-01T13:30:24.478572+0000","last_fullsized":"2024-04-01T13:30:24.478572+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T13:30:25.404306+0000","last_change":"2024-04-01T13:30:24.464855+0000","last_active":"2024-04-01T13:30:25.404306+0000","last_peered":"2024-04-01T13:30:25.404306+0000","last_clean":"2024-04-01T13:30:25.404306+0000","last_became_active":"2024-04-01T13:30:24.458263+0000","last_became_peered":"2024-04-01T13:30:24.458263+0000","last_unstale":"2024-04-01T13:30:25.404306+0000","last_undegraded":"2024-04-01T13:30:25.404306+0000","last_fullsized":"2024-04-01T13:30:25.404306+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.493074+0000","last_change":"2024-04-01T13:30:24.494919+0000","last_active":"2024-04-01T13:30:24.493074+0000","last_peered":"2024-04-01T13:30:24.493074+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:30:24.486637+0000","last_became_peered":"2024-04-01T13:30:24.486637+0000","last_unstale":"2024-04-01T13:30:24.493074+0000","last_undegraded":"2024-04-01T13:30:24.493074+0000","last_fullsized":"2024-04-01T13:30:24.493074+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-01T13:30:25.408144+0000","last_change":"2024-04-01T13:30:24.493976+0000","last_active":"2024-04-01T13:30:25.408144+0000","last_peered":"2024-04-01T13:30:25.408144+0000","last_clean":"2024-04-01T13:30:25.408144+0000","last_became_active":"2024-04-01T13:30:24.483528+0000","last_became_peered":"2024-04-01T13:30:24.483528+0000","last_unstale":"2024-04-01T13:30:25.408144+0000","last_undegraded":"2024-04-01T13:30:25.408144+0000","last_fullsized":"2024-04-01T13:30:25.408144+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-01T13:30:24.962629+0000","last_change":"2024-04-01T13:30:23.446271+0000","last_active":"2024-04-01T13:30:24.962629+0000","last_peered":"2024-04-01T13:30:24.962629+0000","last_clean":"2024-04-01T13:30:24.962629+0000","last_became_active":"2024-04-01T13:30:23.439285+0000","last_became_peered":"2024-04-01T13:30:23.439285+0000","last_unstale":"2024-04-01T13:30:24.962629+0000","last_undegraded":"2024-04-01T13:30:24.962629+0000","last_fullsized":"2024-04-01T13:30:24.962629+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_clean_scrub_stamp":"2024-04-01T13:30:22.378617+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-01T13:30:29.510 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'unique_pool_0' created 2024-04-01T13:30:29.532 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-01T13:30:29.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:29.682+0000 7f1e4f7fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:29.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:29.696+0000 7f1e4f7fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:30.511 INFO:teuthology.orchestra.run.smithi059.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-01T13:30:31.232 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T13:30:31.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:31.362+0000 7f033ff9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:31.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:31.368+0000 7f033ff9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:31.534 DEBUG:teuthology.orchestra.run.smithi059:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0 2024-04-01T13:30:31.567 INFO:tasks.rados.rados.0.smithi059.stdout:adding op weight read -> 100 2024-04-01T13:30:31.567 INFO:tasks.rados.rados.0.smithi059.stdout:adding op weight write -> 50 2024-04-01T13:30:31.567 INFO:tasks.rados.rados.0.smithi059.stdout:adding op weight delete -> 50 2024-04-01T13:30:31.568 INFO:tasks.rados.rados.0.smithi059.stdout:adding op weight snap_create -> 50 2024-04-01T13:30:31.568 INFO:tasks.rados.rados.0.smithi059.stdout:adding op weight snap_remove -> 50 2024-04-01T13:30:31.568 INFO:tasks.rados.rados.0.smithi059.stdout:adding op weight write_excl -> 50 2024-04-01T13:30:31.568 INFO:tasks.rados.rados.0.smithi059.stdout:ceph version 19.0.0-2568-gafa73a9e (afa73a9e21b8fb0610f8af7266f1975be37c0a69) squid (dev) 2024-04-01T13:30:31.568 INFO:tasks.rados.rados.0.smithi059.stdout:Configuration: 2024-04-01T13:30:31.568 INFO:tasks.rados.rados.0.smithi059.stdout: Number of operations: 4000 2024-04-01T13:30:31.568 INFO:tasks.rados.rados.0.smithi059.stdout: Number of objects: 50 2024-04-01T13:30:31.568 INFO:tasks.rados.rados.0.smithi059.stdout: Max in flight operations: 16 2024-04-01T13:30:31.568 INFO:tasks.rados.rados.0.smithi059.stdout: Object size (in bytes): 4000000 2024-04-01T13:30:31.568 INFO:tasks.rados.rados.0.smithi059.stdout: Write stride min: 400000 2024-04-01T13:30:31.568 INFO:tasks.rados.rados.0.smithi059.stdout: Write stride max: 800000 2024-04-01T13:30:32.518 INFO:tasks.rados.rados.0.smithi059.stdout:1: write initial oid 1 2024-04-01T13:30:32.518 INFO:tasks.rados.rados.0.smithi059.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-04-01T13:30:32.523 INFO:tasks.rados.rados.0.smithi059.stdout:1: writing smithi05931183-1 from 416807 to 892056 tid 1 2024-04-01T13:30:32.530 INFO:tasks.rados.rados.0.smithi059.stdout:1: writing smithi05931183-1 from 1542129 to 2016807 tid 2 2024-04-01T13:30:32.530 INFO:tasks.rados.rados.0.smithi059.stdout:2: write initial oid 2 2024-04-01T13:30:32.530 INFO:tasks.rados.rados.0.smithi059.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-04-01T13:30:32.537 INFO:tasks.rados.rados.0.smithi059.stdout:2: writing smithi05931183-2 from 416807 to 892056 tid 1 2024-04-01T13:30:32.545 INFO:tasks.rados.rados.0.smithi059.stdout:2: writing smithi05931183-2 from 1542129 to 2016807 tid 2 2024-04-01T13:30:32.546 INFO:tasks.rados.rados.0.smithi059.stdout:3: write initial oid 3 2024-04-01T13:30:32.546 INFO:tasks.rados.rados.0.smithi059.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-04-01T13:30:32.553 INFO:tasks.rados.rados.0.smithi059.stdout:3: writing smithi05931183-3 from 433614 to 984112 tid 1 2024-04-01T13:30:32.559 INFO:tasks.rados.rados.0.smithi059.stdout:3: writing smithi05931183-3 from 1600611 to 2033614 tid 2 2024-04-01T13:30:32.560 INFO:tasks.rados.rados.0.smithi059.stdout:4: write initial oid 4 2024-04-01T13:30:32.560 INFO:tasks.rados.rados.0.smithi059.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-04-01T13:30:32.568 INFO:tasks.rados.rados.0.smithi059.stdout:4: writing smithi05931183-4 from 450421 to 1076168 tid 1 2024-04-01T13:30:32.576 INFO:tasks.rados.rados.0.smithi059.stdout:4: writing smithi05931183-4 from 1659093 to 2050421 tid 2 2024-04-01T13:30:32.576 INFO:tasks.rados.rados.0.smithi059.stdout:5: write initial oid 5 2024-04-01T13:30:32.576 INFO:tasks.rados.rados.0.smithi059.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-04-01T13:30:32.582 INFO:tasks.rados.rados.0.smithi059.stdout:5: writing smithi05931183-5 from 467228 to 1168224 tid 1 2024-04-01T13:30:32.588 INFO:tasks.rados.rados.0.smithi059.stdout:5: writing smithi05931183-5 from 1717575 to 2067228 tid 2 2024-04-01T13:30:32.588 INFO:tasks.rados.rados.0.smithi059.stdout:6: write initial oid 6 2024-04-01T13:30:32.588 INFO:tasks.rados.rados.0.smithi059.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-04-01T13:30:32.597 INFO:tasks.rados.rados.0.smithi059.stdout:6: writing smithi05931183-6 from 484035 to 1260280 tid 1 2024-04-01T13:30:32.602 INFO:tasks.rados.rados.0.smithi059.stdout:6: writing smithi05931183-6 from 2059704 to 2084035 tid 2 2024-04-01T13:30:32.602 INFO:tasks.rados.rados.0.smithi059.stdout:7: write initial oid 7 2024-04-01T13:30:32.602 INFO:tasks.rados.rados.0.smithi059.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-04-01T13:30:32.609 INFO:tasks.rados.rados.0.smithi059.stdout:7: writing smithi05931183-7 from 500842 to 952336 tid 1 2024-04-01T13:30:32.617 INFO:tasks.rados.rados.0.smithi059.stdout:7: writing smithi05931183-7 from 1718186 to 2100842 tid 2 2024-04-01T13:30:32.617 INFO:tasks.rados.rados.0.smithi059.stdout:8: write initial oid 8 2024-04-01T13:30:32.617 INFO:tasks.rados.rados.0.smithi059.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-04-01T13:30:32.625 INFO:tasks.rados.rados.0.smithi059.stdout:8: writing smithi05931183-8 from 517649 to 1044392 tid 1 2024-04-01T13:30:32.632 INFO:tasks.rados.rados.0.smithi059.stdout:8: writing smithi05931183-8 from 1776668 to 2117649 tid 2 2024-04-01T13:30:32.633 INFO:tasks.rados.rados.0.smithi059.stdout:9: write initial oid 9 2024-04-01T13:30:32.633 INFO:tasks.rados.rados.0.smithi059.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-04-01T13:30:32.640 INFO:tasks.rados.rados.0.smithi059.stdout:9: writing smithi05931183-9 from 534456 to 1252801 tid 1 2024-04-01T13:30:32.647 INFO:tasks.rados.rados.0.smithi059.stdout:9: writing smithi05931183-9 from 1951503 to 2134456 tid 2 2024-04-01T13:30:32.647 INFO:tasks.rados.rados.0.smithi059.stdout:10: write initial oid 10 2024-04-01T13:30:32.647 INFO:tasks.rados.rados.0.smithi059.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-04-01T13:30:32.656 INFO:tasks.rados.rados.0.smithi059.stdout:10: writing smithi05931183-10 from 551263 to 1344857 tid 1 2024-04-01T13:30:32.661 INFO:tasks.rados.rados.0.smithi059.stdout:10: writing smithi05931183-10 from 1893632 to 2151263 tid 2 2024-04-01T13:30:32.661 INFO:tasks.rados.rados.0.smithi059.stdout:11: write initial oid 11 2024-04-01T13:30:32.661 INFO:tasks.rados.rados.0.smithi059.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-04-01T13:30:32.667 INFO:tasks.rados.rados.0.smithi059.stdout:11: writing smithi05931183-11 from 568070 to 1036913 tid 1 2024-04-01T13:30:32.675 INFO:tasks.rados.rados.0.smithi059.stdout:11: writing smithi05931183-11 from 1552114 to 2168070 tid 2 2024-04-01T13:30:32.675 INFO:tasks.rados.rados.0.smithi059.stdout:12: write initial oid 12 2024-04-01T13:30:32.675 INFO:tasks.rados.rados.0.smithi059.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-04-01T13:30:32.682 INFO:tasks.rados.rados.0.smithi059.stdout:12: writing smithi05931183-12 from 584877 to 1128969 tid 1 2024-04-01T13:30:32.691 INFO:tasks.rados.rados.0.smithi059.stdout:12: writing smithi05931183-12 from 1610596 to 2172599 tid 2 2024-04-01T13:30:32.692 INFO:tasks.rados.rados.0.smithi059.stdout:13: write initial oid 13 2024-04-01T13:30:32.692 INFO:tasks.rados.rados.0.smithi059.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-04-01T13:30:32.701 INFO:tasks.rados.rados.0.smithi059.stdout:13: writing smithi05931183-13 from 601684 to 1221025 tid 1 2024-04-01T13:30:32.708 INFO:tasks.rados.rados.0.smithi059.stdout:13: writing smithi05931183-13 from 1669078 to 2201684 tid 2 2024-04-01T13:30:32.708 INFO:tasks.rados.rados.0.smithi059.stdout:14: write initial oid 14 2024-04-01T13:30:32.708 INFO:tasks.rados.rados.0.smithi059.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-04-01T13:30:32.716 INFO:tasks.rados.rados.0.smithi059.stdout:14: writing smithi05931183-14 from 618491 to 1313081 tid 1 2024-04-01T13:30:32.721 INFO:tasks.rados.rados.0.smithi059.stdout:14: writing smithi05931183-14 from 2011207 to 2218491 tid 2 2024-04-01T13:30:32.721 INFO:tasks.rados.rados.0.smithi059.stdout:15: write initial oid 15 2024-04-01T13:30:32.721 INFO:tasks.rados.rados.0.smithi059.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-04-01T13:30:32.732 INFO:tasks.rados.rados.0.smithi059.stdout:15: writing smithi05931183-15 from 635298 to 1405137 tid 1 2024-04-01T13:30:32.738 INFO:tasks.rados.rados.0.smithi059.stdout:15: writing smithi05931183-15 from 2069689 to 2235298 tid 2 2024-04-01T13:30:32.738 INFO:tasks.rados.rados.0.smithi059.stdout:16: write initial oid 16 2024-04-01T13:30:32.738 INFO:tasks.rados.rados.0.smithi059.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-04-01T13:30:32.745 INFO:tasks.rados.rados.0.smithi059.stdout:16: writing smithi05931183-16 from 652105 to 1097193 tid 1 2024-04-01T13:30:32.754 INFO:tasks.rados.rados.0.smithi059.stdout:16: writing smithi05931183-16 from 1728171 to 2181159 tid 2 2024-04-01T13:30:32.754 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:32.754 INFO:tasks.rados.rados.0.smithi059.stdout:1: finishing write tid 1 to smithi05931183-1 2024-04-01T13:30:32.754 INFO:tasks.rados.rados.0.smithi059.stdout:3: finishing write tid 1 to smithi05931183-3 2024-04-01T13:30:32.754 INFO:tasks.rados.rados.0.smithi059.stdout:1: finishing write tid 2 to smithi05931183-1 2024-04-01T13:30:32.754 INFO:tasks.rados.rados.0.smithi059.stdout:1: finishing write tid 3 to smithi05931183-1 2024-04-01T13:30:32.754 INFO:tasks.rados.rados.0.smithi059.stdout:1: finishing write tid 4 to smithi05931183-1 2024-04-01T13:30:32.754 INFO:tasks.rados.rados.0.smithi059.stdout:1: oid 1 updating version 0 to 3 2024-04-01T13:30:32.754 INFO:tasks.rados.rados.0.smithi059.stdout:1: oid 1 version 3 is already newer than 1 2024-04-01T13:30:32.754 INFO:tasks.rados.rados.0.smithi059.stdout:1: oid 1 version 3 is already newer than 3 2024-04-01T13:30:32.754 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-01T13:30:32.755 INFO:tasks.rados.rados.0.smithi059.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-04-01T13:30:32.755 INFO:tasks.rados.rados.0.smithi059.stdout:3: finishing write tid 2 to smithi05931183-3 2024-04-01T13:30:32.755 INFO:tasks.rados.rados.0.smithi059.stdout:4: finishing write tid 1 to smithi05931183-4 2024-04-01T13:30:32.755 INFO:tasks.rados.rados.0.smithi059.stdout:5: finishing write tid 1 to smithi05931183-5 2024-04-01T13:30:32.755 INFO:tasks.rados.rados.0.smithi059.stdout:1: done (15 left) 2024-04-01T13:30:32.755 INFO:tasks.rados.rados.0.smithi059.stdout:17: write initial oid 17 2024-04-01T13:30:32.755 INFO:tasks.rados.rados.0.smithi059.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-04-01T13:30:32.762 INFO:tasks.rados.rados.0.smithi059.stdout:17: writing smithi05931183-17 from 668912 to 1305602 tid 1 2024-04-01T13:30:32.769 INFO:tasks.rados.rados.0.smithi059.stdout:17: writing smithi05931183-17 from 1903006 to 2268912 tid 2 2024-04-01T13:30:32.769 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:32.769 INFO:tasks.rados.rados.0.smithi059.stdout:2: finishing write tid 1 to smithi05931183-2 2024-04-01T13:30:32.769 INFO:tasks.rados.rados.0.smithi059.stdout:7: finishing write tid 1 to smithi05931183-7 2024-04-01T13:30:32.769 INFO:tasks.rados.rados.0.smithi059.stdout:8: finishing write tid 1 to smithi05931183-8 2024-04-01T13:30:32.769 INFO:tasks.rados.rados.0.smithi059.stdout:3: finishing write tid 3 to smithi05931183-3 2024-04-01T13:30:32.769 INFO:tasks.rados.rados.0.smithi059.stdout:3: finishing write tid 4 to smithi05931183-3 2024-04-01T13:30:32.769 INFO:tasks.rados.rados.0.smithi059.stdout:3: oid 3 updating version 0 to 1 2024-04-01T13:30:32.769 INFO:tasks.rados.rados.0.smithi059.stdout:3: oid 3 updating version 1 to 2 2024-04-01T13:30:32.769 INFO:tasks.rados.rados.0.smithi059.stdout:3: oid 3 updating version 2 to 3 2024-04-01T13:30:32.769 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-01T13:30:32.769 INFO:tasks.rados.rados.0.smithi059.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-04-01T13:30:32.769 INFO:tasks.rados.rados.0.smithi059.stdout:6: finishing write tid 1 to smithi05931183-6 2024-04-01T13:30:32.770 INFO:tasks.rados.rados.0.smithi059.stdout:4: finishing write tid 2 to smithi05931183-4 2024-04-01T13:30:32.770 INFO:tasks.rados.rados.0.smithi059.stdout:2: finishing write tid 2 to smithi05931183-2 2024-04-01T13:30:32.770 INFO:tasks.rados.rados.0.smithi059.stdout:3: done (15 left) 2024-04-01T13:30:32.770 INFO:tasks.rados.rados.0.smithi059.stdout:18: write initial oid 18 2024-04-01T13:30:32.770 INFO:tasks.rados.rados.0.smithi059.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-04-01T13:30:32.777 INFO:tasks.rados.rados.0.smithi059.stdout:18: writing smithi05931183-18 from 685719 to 1397658 tid 1 2024-04-01T13:30:32.784 INFO:tasks.rados.rados.0.smithi059.stdout:18: writing smithi05931183-18 from 1845135 to 2285719 tid 2 2024-04-01T13:30:32.784 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:32.784 INFO:tasks.rados.rados.0.smithi059.stdout:7: finishing write tid 2 to smithi05931183-7 2024-04-01T13:30:32.784 INFO:tasks.rados.rados.0.smithi059.stdout:8: finishing write tid 2 to smithi05931183-8 2024-04-01T13:30:32.784 INFO:tasks.rados.rados.0.smithi059.stdout:9: finishing write tid 1 to smithi05931183-9 2024-04-01T13:30:32.784 INFO:tasks.rados.rados.0.smithi059.stdout:11: finishing write tid 1 to smithi05931183-11 2024-04-01T13:30:32.784 INFO:tasks.rados.rados.0.smithi059.stdout:5: finishing write tid 2 to smithi05931183-5 2024-04-01T13:30:32.784 INFO:tasks.rados.rados.0.smithi059.stdout:10: finishing write tid 1 to smithi05931183-10 2024-04-01T13:30:32.784 INFO:tasks.rados.rados.0.smithi059.stdout:6: finishing write tid 2 to smithi05931183-6 2024-04-01T13:30:32.784 INFO:tasks.rados.rados.0.smithi059.stdout:4: finishing write tid 3 to smithi05931183-4 2024-04-01T13:30:32.784 INFO:tasks.rados.rados.0.smithi059.stdout:2: finishing write tid 3 to smithi05931183-2 2024-04-01T13:30:32.784 INFO:tasks.rados.rados.0.smithi059.stdout:7: finishing write tid 3 to smithi05931183-7 2024-04-01T13:30:32.784 INFO:tasks.rados.rados.0.smithi059.stdout:2: finishing write tid 4 to smithi05931183-2 2024-04-01T13:30:32.784 INFO:tasks.rados.rados.0.smithi059.stdout:2: oid 2 updating version 0 to 4 2024-04-01T13:30:32.785 INFO:tasks.rados.rados.0.smithi059.stdout:2: oid 2 updating version 4 to 5 2024-04-01T13:30:32.785 INFO:tasks.rados.rados.0.smithi059.stdout:2: oid 2 updating version 5 to 6 2024-04-01T13:30:32.785 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 2 v 6 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-01T13:30:32.785 INFO:tasks.rados.rados.0.smithi059.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-04-01T13:30:32.785 INFO:tasks.rados.rados.0.smithi059.stdout:7: finishing write tid 4 to smithi05931183-7 2024-04-01T13:30:32.785 INFO:tasks.rados.rados.0.smithi059.stdout:7: oid 7 updating version 0 to 1 2024-04-01T13:30:32.785 INFO:tasks.rados.rados.0.smithi059.stdout:7: oid 7 updating version 1 to 3 2024-04-01T13:30:32.785 INFO:tasks.rados.rados.0.smithi059.stdout:7: oid 7 version 3 is already newer than 2 2024-04-01T13:30:32.785 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-01T13:30:32.785 INFO:tasks.rados.rados.0.smithi059.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-04-01T13:30:32.785 INFO:tasks.rados.rados.0.smithi059.stdout:4: finishing write tid 4 to smithi05931183-4 2024-04-01T13:30:32.785 INFO:tasks.rados.rados.0.smithi059.stdout:4: oid 4 updating version 0 to 3 2024-04-01T13:30:32.785 INFO:tasks.rados.rados.0.smithi059.stdout:4: oid 4 version 3 is already newer than 1 2024-04-01T13:30:32.785 INFO:tasks.rados.rados.0.smithi059.stdout:4: oid 4 version 3 is already newer than 2 2024-04-01T13:30:32.785 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-01T13:30:32.786 INFO:tasks.rados.rados.0.smithi059.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-04-01T13:30:32.786 INFO:tasks.rados.rados.0.smithi059.stdout:2: done (15 left) 2024-04-01T13:30:32.786 INFO:tasks.rados.rados.0.smithi059.stdout:4: done (14 left) 2024-04-01T13:30:32.787 INFO:tasks.rados.rados.0.smithi059.stdout:7: done (13 left) 2024-04-01T13:30:32.787 INFO:tasks.rados.rados.0.smithi059.stdout:19: write initial oid 19 2024-04-01T13:30:32.787 INFO:tasks.rados.rados.0.smithi059.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-04-01T13:30:32.792 INFO:tasks.rados.rados.0.smithi059.stdout:19: writing smithi05931183-19 from 702526 to 1489714 tid 1 2024-04-01T13:30:32.797 INFO:tasks.rados.rados.0.smithi059.stdout:19: writing smithi05931183-19 from 1903617 to 2302526 tid 2 2024-04-01T13:30:32.797 INFO:tasks.rados.rados.0.smithi059.stdout:20: write initial oid 20 2024-04-01T13:30:32.797 INFO:tasks.rados.rados.0.smithi059.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-04-01T13:30:32.805 INFO:tasks.rados.rados.0.smithi059.stdout:20: writing smithi05931183-20 from 719333 to 1181770 tid 1 2024-04-01T13:30:32.814 INFO:tasks.rados.rados.0.smithi059.stdout:20: writing smithi05931183-20 from 1962099 to 2319333 tid 2 2024-04-01T13:30:32.814 INFO:tasks.rados.rados.0.smithi059.stdout:21: write initial oid 21 2024-04-01T13:30:32.814 INFO:tasks.rados.rados.0.smithi059.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-04-01T13:30:32.821 INFO:tasks.rados.rados.0.smithi059.stdout:21: writing smithi05931183-21 from 736140 to 1273826 tid 1 2024-04-01T13:30:32.828 INFO:tasks.rados.rados.0.smithi059.stdout:21: writing smithi05931183-21 from 2020581 to 2336140 tid 2 2024-04-01T13:30:32.829 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:32.829 INFO:tasks.rados.rados.0.smithi059.stdout:8: finishing write tid 3 to smithi05931183-8 2024-04-01T13:30:32.829 INFO:tasks.rados.rados.0.smithi059.stdout:8: finishing write tid 4 to smithi05931183-8 2024-04-01T13:30:32.829 INFO:tasks.rados.rados.0.smithi059.stdout:8: oid 8 updating version 0 to 2 2024-04-01T13:30:32.829 INFO:tasks.rados.rados.0.smithi059.stdout:8: oid 8 version 2 is already newer than 1 2024-04-01T13:30:32.829 INFO:tasks.rados.rados.0.smithi059.stdout:8: oid 8 updating version 2 to 3 2024-04-01T13:30:32.829 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-01T13:30:32.829 INFO:tasks.rados.rados.0.smithi059.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-04-01T13:30:32.829 INFO:tasks.rados.rados.0.smithi059.stdout:15: finishing write tid 1 to smithi05931183-15 2024-04-01T13:30:32.829 INFO:tasks.rados.rados.0.smithi059.stdout:9: finishing write tid 2 to smithi05931183-9 2024-04-01T13:30:32.829 INFO:tasks.rados.rados.0.smithi059.stdout:10: finishing write tid 2 to smithi05931183-10 2024-04-01T13:30:32.829 INFO:tasks.rados.rados.0.smithi059.stdout:6: finishing write tid 3 to smithi05931183-6 2024-04-01T13:30:32.829 INFO:tasks.rados.rados.0.smithi059.stdout:11: finishing write tid 2 to smithi05931183-11 2024-04-01T13:30:32.830 INFO:tasks.rados.rados.0.smithi059.stdout:9: finishing write tid 3 to smithi05931183-9 2024-04-01T13:30:32.830 INFO:tasks.rados.rados.0.smithi059.stdout:9: finishing write tid 4 to smithi05931183-9 2024-04-01T13:30:32.830 INFO:tasks.rados.rados.0.smithi059.stdout:9: oid 9 updating version 0 to 1 2024-04-01T13:30:32.830 INFO:tasks.rados.rados.0.smithi059.stdout:9: oid 9 updating version 1 to 3 2024-04-01T13:30:32.830 INFO:tasks.rados.rados.0.smithi059.stdout:9: oid 9 version 3 is already newer than 3 2024-04-01T13:30:32.830 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-01T13:30:32.830 INFO:tasks.rados.rados.0.smithi059.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-04-01T13:30:32.830 INFO:tasks.rados.rados.0.smithi059.stdout:5: finishing write tid 3 to smithi05931183-5 2024-04-01T13:30:32.830 INFO:tasks.rados.rados.0.smithi059.stdout:5: finishing write tid 4 to smithi05931183-5 2024-04-01T13:30:32.830 INFO:tasks.rados.rados.0.smithi059.stdout:5: oid 5 updating version 0 to 1 2024-04-01T13:30:32.830 INFO:tasks.rados.rados.0.smithi059.stdout:5: oid 5 updating version 1 to 2 2024-04-01T13:30:32.830 INFO:tasks.rados.rados.0.smithi059.stdout:5: oid 5 updating version 2 to 3 2024-04-01T13:30:32.830 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-01T13:30:32.830 INFO:tasks.rados.rados.0.smithi059.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-04-01T13:30:32.830 INFO:tasks.rados.rados.0.smithi059.stdout:5: done (15 left) 2024-04-01T13:30:32.831 INFO:tasks.rados.rados.0.smithi059.stdout:8: done (14 left) 2024-04-01T13:30:32.831 INFO:tasks.rados.rados.0.smithi059.stdout:9: done (13 left) 2024-04-01T13:30:32.831 INFO:tasks.rados.rados.0.smithi059.stdout:22: write initial oid 22 2024-04-01T13:30:32.831 INFO:tasks.rados.rados.0.smithi059.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-04-01T13:30:32.835 INFO:tasks.rados.rados.0.smithi059.stdout:22: writing smithi05931183-22 from 752947 to 1365882 tid 1 2024-04-01T13:30:32.844 INFO:tasks.rados.rados.0.smithi059.stdout:22: writing smithi05931183-22 from 1962710 to 2352947 tid 2 2024-04-01T13:30:32.844 INFO:tasks.rados.rados.0.smithi059.stdout:23: write initial oid 23 2024-04-01T13:30:32.844 INFO:tasks.rados.rados.0.smithi059.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-04-01T13:30:32.850 INFO:tasks.rados.rados.0.smithi059.stdout:23: writing smithi05931183-23 from 769754 to 1457938 tid 1 2024-04-01T13:30:32.858 INFO:tasks.rados.rados.0.smithi059.stdout:23: writing smithi05931183-23 from 2021192 to 2369754 tid 2 2024-04-01T13:30:32.858 INFO:tasks.rados.rados.0.smithi059.stdout:24: write initial oid 24 2024-04-01T13:30:32.858 INFO:tasks.rados.rados.0.smithi059.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-04-01T13:30:32.865 INFO:tasks.rados.rados.0.smithi059.stdout:24: writing smithi05931183-24 from 786561 to 1266347 tid 1 2024-04-01T13:30:32.874 INFO:tasks.rados.rados.0.smithi059.stdout:24: writing smithi05931183-24 from 1796027 to 2263691 tid 2 2024-04-01T13:30:32.874 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:32.874 INFO:tasks.rados.rados.0.smithi059.stdout:6: finishing write tid 4 to smithi05931183-6 2024-04-01T13:30:32.874 INFO:tasks.rados.rados.0.smithi059.stdout:6: oid 6 updating version 0 to 2 2024-04-01T13:30:32.874 INFO:tasks.rados.rados.0.smithi059.stdout:6: oid 6 version 2 is already newer than 1 2024-04-01T13:30:32.874 INFO:tasks.rados.rados.0.smithi059.stdout:6: oid 6 updating version 2 to 3 2024-04-01T13:30:32.874 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-01T13:30:32.874 INFO:tasks.rados.rados.0.smithi059.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-04-01T13:30:32.874 INFO:tasks.rados.rados.0.smithi059.stdout:6: done (15 left) 2024-04-01T13:30:32.874 INFO:tasks.rados.rados.0.smithi059.stdout:25: write initial oid 25 2024-04-01T13:30:32.875 INFO:tasks.rados.rados.0.smithi059.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-04-01T13:30:32.881 INFO:tasks.rados.rados.0.smithi059.stdout:25: writing smithi05931183-25 from 403368 to 958403 tid 1 2024-04-01T13:30:32.891 INFO:tasks.rados.rados.0.smithi059.stdout:25: writing smithi05931183-25 from 1454509 to 2182184 tid 2 2024-04-01T13:30:32.891 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:32.891 INFO:tasks.rados.rados.0.smithi059.stdout:15: finishing write tid 2 to smithi05931183-15 2024-04-01T13:30:32.891 INFO:tasks.rados.rados.0.smithi059.stdout:18: finishing write tid 1 to smithi05931183-18 2024-04-01T13:30:32.891 INFO:tasks.rados.rados.0.smithi059.stdout:11: finishing write tid 3 to smithi05931183-11 2024-04-01T13:30:32.891 INFO:tasks.rados.rados.0.smithi059.stdout:11: finishing write tid 4 to smithi05931183-11 2024-04-01T13:30:32.891 INFO:tasks.rados.rados.0.smithi059.stdout:11: oid 11 updating version 0 to 2 2024-04-01T13:30:32.891 INFO:tasks.rados.rados.0.smithi059.stdout:11: oid 11 version 2 is already newer than 1 2024-04-01T13:30:32.891 INFO:tasks.rados.rados.0.smithi059.stdout:11: oid 11 updating version 2 to 3 2024-04-01T13:30:32.891 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-01T13:30:32.891 INFO:tasks.rados.rados.0.smithi059.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-04-01T13:30:32.892 INFO:tasks.rados.rados.0.smithi059.stdout:10: finishing write tid 3 to smithi05931183-10 2024-04-01T13:30:32.892 INFO:tasks.rados.rados.0.smithi059.stdout:12: finishing write tid 1 to smithi05931183-12 2024-04-01T13:30:32.892 INFO:tasks.rados.rados.0.smithi059.stdout:11: done (15 left) 2024-04-01T13:30:32.892 INFO:tasks.rados.rados.0.smithi059.stdout:26: write initial oid 26 2024-04-01T13:30:32.892 INFO:tasks.rados.rados.0.smithi059.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-04-01T13:30:32.896 INFO:tasks.rados.rados.0.smithi059.stdout:26: writing smithi05931183-26 from 420175 to 1050459 tid 1 2024-04-01T13:30:32.906 INFO:tasks.rados.rados.0.smithi059.stdout:26: writing smithi05931183-26 from 1796638 to 2267971 tid 2 2024-04-01T13:30:32.906 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:32.906 INFO:tasks.rados.rados.0.smithi059.stdout:10: finishing write tid 4 to smithi05931183-10 2024-04-01T13:30:32.906 INFO:tasks.rados.rados.0.smithi059.stdout:10: oid 10 updating version 0 to 2 2024-04-01T13:30:32.906 INFO:tasks.rados.rados.0.smithi059.stdout:10: oid 10 version 2 is already newer than 1 2024-04-01T13:30:32.906 INFO:tasks.rados.rados.0.smithi059.stdout:10: oid 10 updating version 2 to 3 2024-04-01T13:30:32.906 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-01T13:30:32.906 INFO:tasks.rados.rados.0.smithi059.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-04-01T13:30:32.906 INFO:tasks.rados.rados.0.smithi059.stdout:13: finishing write tid 1 to smithi05931183-13 2024-04-01T13:30:32.906 INFO:tasks.rados.rados.0.smithi059.stdout:10: done (15 left) 2024-04-01T13:30:32.906 INFO:tasks.rados.rados.0.smithi059.stdout:27: write initial oid 27 2024-04-01T13:30:32.906 INFO:tasks.rados.rados.0.smithi059.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-04-01T13:30:32.911 INFO:tasks.rados.rados.0.smithi059.stdout:27: writing smithi05931183-27 from 436982 to 1142515 tid 1 2024-04-01T13:30:32.921 INFO:tasks.rados.rados.0.smithi059.stdout:27: writing smithi05931183-27 from 1855120 to 2436982 tid 2 2024-04-01T13:30:32.921 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:32.921 INFO:tasks.rados.rados.0.smithi059.stdout:17: finishing write tid 1 to smithi05931183-17 2024-04-01T13:30:32.921 INFO:tasks.rados.rados.0.smithi059.stdout:14: finishing write tid 1 to smithi05931183-14 2024-04-01T13:30:32.953 INFO:tasks.rados.rados.0.smithi059.stdout:15: finishing write tid 3 to smithi05931183-15 2024-04-01T13:30:32.954 INFO:tasks.rados.rados.0.smithi059.stdout:15: finishing write tid 4 to smithi05931183-15 2024-04-01T13:30:32.954 INFO:tasks.rados.rados.0.smithi059.stdout:15: oid 15 updating version 0 to 2 2024-04-01T13:30:32.955 INFO:tasks.rados.rados.0.smithi059.stdout:15: oid 15 version 2 is already newer than 1 2024-04-01T13:30:32.955 INFO:tasks.rados.rados.0.smithi059.stdout:15: oid 15 updating version 2 to 3 2024-04-01T13:30:32.955 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-01T13:30:32.955 INFO:tasks.rados.rados.0.smithi059.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-04-01T13:30:32.955 INFO:tasks.rados.rados.0.smithi059.stdout:15: done (15 left) 2024-04-01T13:30:32.955 INFO:tasks.rados.rados.0.smithi059.stdout:28: write initial oid 28 2024-04-01T13:30:32.955 INFO:tasks.rados.rados.0.smithi059.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-04-01T13:30:32.961 INFO:tasks.rados.rados.0.smithi059.stdout:28: writing smithi05931183-28 from 453789 to 1234571 tid 1 2024-04-01T13:30:32.968 INFO:tasks.rados.rados.0.smithi059.stdout:28: writing smithi05931183-28 from 1913602 to 2388604 tid 2 2024-04-01T13:30:32.968 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:32.968 INFO:tasks.rados.rados.0.smithi059.stdout:22: finishing write tid 1 to smithi05931183-22 2024-04-01T13:30:32.968 INFO:tasks.rados.rados.0.smithi059.stdout:24: finishing write tid 1 to smithi05931183-24 2024-04-01T13:30:32.968 INFO:tasks.rados.rados.0.smithi059.stdout:18: finishing write tid 2 to smithi05931183-18 2024-04-01T13:30:32.972 INFO:tasks.rados.rados.0.smithi059.stdout:16: finishing write tid 1 to smithi05931183-16 2024-04-01T13:30:32.975 INFO:tasks.rados.rados.0.smithi059.stdout:17: finishing write tid 2 to smithi05931183-17 2024-04-01T13:30:32.978 INFO:tasks.rados.rados.0.smithi059.stdout:12: finishing write tid 2 to smithi05931183-12 2024-04-01T13:30:32.986 INFO:tasks.rados.rados.0.smithi059.stdout:17: finishing write tid 3 to smithi05931183-17 2024-04-01T13:30:32.987 INFO:tasks.rados.rados.0.smithi059.stdout:14: finishing write tid 2 to smithi05931183-14 2024-04-01T13:30:32.991 INFO:tasks.rados.rados.0.smithi059.stdout:14: finishing write tid 3 to smithi05931183-14 2024-04-01T13:30:32.993 INFO:tasks.rados.rados.0.smithi059.stdout:14: finishing write tid 4 to smithi05931183-14 2024-04-01T13:30:32.993 INFO:tasks.rados.rados.0.smithi059.stdout:14: oid 14 updating version 0 to 6 2024-04-01T13:30:32.994 INFO:tasks.rados.rados.0.smithi059.stdout:14: oid 14 version 6 is already newer than 4 2024-04-01T13:30:32.994 INFO:tasks.rados.rados.0.smithi059.stdout:14: oid 14 version 6 is already newer than 6 2024-04-01T13:30:32.994 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-01T13:30:32.994 INFO:tasks.rados.rados.0.smithi059.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-04-01T13:30:32.994 INFO:tasks.rados.rados.0.smithi059.stdout:14: done (15 left) 2024-04-01T13:30:32.994 INFO:tasks.rados.rados.0.smithi059.stdout:29: write initial oid 29 2024-04-01T13:30:32.994 INFO:tasks.rados.rados.0.smithi059.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-04-01T13:30:32.999 INFO:tasks.rados.rados.0.smithi059.stdout:29: writing smithi05931183-29 from 470596 to 926627 tid 1 2024-04-01T13:30:33.007 INFO:tasks.rados.rados.0.smithi059.stdout:29: writing smithi05931183-29 from 1572084 to 2190744 tid 2 2024-04-01T13:30:33.007 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.007 INFO:tasks.rados.rados.0.smithi059.stdout:17: finishing write tid 4 to smithi05931183-17 2024-04-01T13:30:33.007 INFO:tasks.rados.rados.0.smithi059.stdout:17: oid 17 updating version 0 to 4 2024-04-01T13:30:33.007 INFO:tasks.rados.rados.0.smithi059.stdout:17: oid 17 updating version 4 to 6 2024-04-01T13:30:33.007 INFO:tasks.rados.rados.0.smithi059.stdout:17: oid 17 version 6 is already newer than 6 2024-04-01T13:30:33.007 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-01T13:30:33.007 INFO:tasks.rados.rados.0.smithi059.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-04-01T13:30:33.007 INFO:tasks.rados.rados.0.smithi059.stdout:17: done (15 left) 2024-04-01T13:30:33.007 INFO:tasks.rados.rados.0.smithi059.stdout:30: write initial oid 30 2024-04-01T13:30:33.007 INFO:tasks.rados.rados.0.smithi059.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-04-01T13:30:33.012 INFO:tasks.rados.rados.0.smithi059.stdout:30: writing smithi05931183-30 from 487403 to 1018683 tid 1 2024-04-01T13:30:33.017 INFO:tasks.rados.rados.0.smithi059.stdout:30: writing smithi05931183-30 from 1514213 to 1992884 tid 2 2024-04-01T13:30:33.017 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.017 INFO:tasks.rados.rados.0.smithi059.stdout:13: finishing write tid 2 to smithi05931183-13 2024-04-01T13:30:33.024 INFO:tasks.rados.rados.0.smithi059.stdout:27: finishing write tid 1 to smithi05931183-27 2024-04-01T13:30:33.035 INFO:tasks.rados.rados.0.smithi059.stdout:24: finishing write tid 2 to smithi05931183-24 2024-04-01T13:30:33.044 INFO:tasks.rados.rados.0.smithi059.stdout:16: finishing write tid 2 to smithi05931183-16 2024-04-01T13:30:33.047 INFO:tasks.rados.rados.0.smithi059.stdout:22: finishing write tid 2 to smithi05931183-22 2024-04-01T13:30:33.051 INFO:tasks.rados.rados.0.smithi059.stdout:12: finishing write tid 3 to smithi05931183-12 2024-04-01T13:30:33.052 INFO:tasks.rados.rados.0.smithi059.stdout:16: finishing write tid 3 to smithi05931183-16 2024-04-01T13:30:33.053 INFO:tasks.rados.rados.0.smithi059.stdout:18: finishing write tid 3 to smithi05931183-18 2024-04-01T13:30:33.055 INFO:tasks.rados.rados.0.smithi059.stdout:21: finishing write tid 1 to smithi05931183-21 2024-04-01T13:30:33.057 INFO:tasks.rados.rados.0.smithi059.stdout:18: finishing write tid 4 to smithi05931183-18 2024-04-01T13:30:33.058 INFO:tasks.rados.rados.0.smithi059.stdout:18: oid 18 updating version 0 to 2 2024-04-01T13:30:33.058 INFO:tasks.rados.rados.0.smithi059.stdout:18: oid 18 updating version 2 to 3 2024-04-01T13:30:33.058 INFO:tasks.rados.rados.0.smithi059.stdout:18: oid 18 version 3 is already newer than 1 2024-04-01T13:30:33.058 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 18 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-01T13:30:33.058 INFO:tasks.rados.rados.0.smithi059.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-04-01T13:30:33.058 INFO:tasks.rados.rados.0.smithi059.stdout:18: done (15 left) 2024-04-01T13:30:33.058 INFO:tasks.rados.rados.0.smithi059.stdout:31: write initial oid 31 2024-04-01T13:30:33.058 INFO:tasks.rados.rados.0.smithi059.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-04-01T13:30:33.064 INFO:tasks.rados.rados.0.smithi059.stdout:31: writing smithi05931183-31 from 504210 to 1110739 tid 1 2024-04-01T13:30:33.071 INFO:tasks.rados.rados.0.smithi059.stdout:31: writing smithi05931183-31 from 1572695 to 2195024 tid 2 2024-04-01T13:30:33.071 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.071 INFO:tasks.rados.rados.0.smithi059.stdout:12: finishing write tid 4 to smithi05931183-12 2024-04-01T13:30:33.071 INFO:tasks.rados.rados.0.smithi059.stdout:12: oid 12 updating version 0 to 4 2024-04-01T13:30:33.071 INFO:tasks.rados.rados.0.smithi059.stdout:12: oid 12 updating version 4 to 5 2024-04-01T13:30:33.071 INFO:tasks.rados.rados.0.smithi059.stdout:12: oid 12 updating version 5 to 6 2024-04-01T13:30:33.071 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 12 v 6 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-01T13:30:33.071 INFO:tasks.rados.rados.0.smithi059.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-04-01T13:30:33.071 INFO:tasks.rados.rados.0.smithi059.stdout:12: done (15 left) 2024-04-01T13:30:33.072 INFO:tasks.rados.rados.0.smithi059.stdout:32: write initial oid 32 2024-04-01T13:30:33.072 INFO:tasks.rados.rados.0.smithi059.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-04-01T13:30:33.081 INFO:tasks.rados.rados.0.smithi059.stdout:32: writing smithi05931183-32 from 521017 to 1319148 tid 1 2024-04-01T13:30:33.087 INFO:tasks.rados.rados.0.smithi059.stdout:32: writing smithi05931183-32 from 1747530 to 2229870 tid 2 2024-04-01T13:30:33.088 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.088 INFO:tasks.rados.rados.0.smithi059.stdout:16: finishing write tid 4 to smithi05931183-16 2024-04-01T13:30:33.088 INFO:tasks.rados.rados.0.smithi059.stdout:16: oid 16 updating version 0 to 6 2024-04-01T13:30:33.088 INFO:tasks.rados.rados.0.smithi059.stdout:16: oid 16 version 6 is already newer than 6 2024-04-01T13:30:33.088 INFO:tasks.rados.rados.0.smithi059.stdout:16: oid 16 version 6 is already newer than 4 2024-04-01T13:30:33.088 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-01T13:30:33.088 INFO:tasks.rados.rados.0.smithi059.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-04-01T13:30:33.088 INFO:tasks.rados.rados.0.smithi059.stdout:24: finishing write tid 3 to smithi05931183-24 2024-04-01T13:30:33.088 INFO:tasks.rados.rados.0.smithi059.stdout:24: finishing write tid 4 to smithi05931183-24 2024-04-01T13:30:33.088 INFO:tasks.rados.rados.0.smithi059.stdout:24: oid 24 updating version 0 to 4 2024-04-01T13:30:33.088 INFO:tasks.rados.rados.0.smithi059.stdout:24: oid 24 updating version 4 to 5 2024-04-01T13:30:33.088 INFO:tasks.rados.rados.0.smithi059.stdout:24: oid 24 updating version 5 to 6 2024-04-01T13:30:33.088 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 24 v 6 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-01T13:30:33.088 INFO:tasks.rados.rados.0.smithi059.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-04-01T13:30:33.089 INFO:tasks.rados.rados.0.smithi059.stdout:16: done (15 left) 2024-04-01T13:30:33.089 INFO:tasks.rados.rados.0.smithi059.stdout:24: done (14 left) 2024-04-01T13:30:33.089 INFO:tasks.rados.rados.0.smithi059.stdout:33: write initial oid 33 2024-04-01T13:30:33.089 INFO:tasks.rados.rados.0.smithi059.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-04-01T13:30:33.092 INFO:tasks.rados.rados.0.smithi059.stdout:33: writing smithi05931183-33 from 537824 to 1011204 tid 1 2024-04-01T13:30:33.100 INFO:tasks.rados.rados.0.smithi059.stdout:33: writing smithi05931183-33 from 1806012 to 2432010 tid 2 2024-04-01T13:30:33.100 INFO:tasks.rados.rados.0.smithi059.stdout:34: write initial oid 34 2024-04-01T13:30:33.100 INFO:tasks.rados.rados.0.smithi059.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-04-01T13:30:33.106 INFO:tasks.rados.rados.0.smithi059.stdout:34: writing smithi05931183-34 from 554631 to 1103260 tid 1 2024-04-01T13:30:33.114 INFO:tasks.rados.rados.0.smithi059.stdout:34: writing smithi05931183-34 from 1748141 to 2234150 tid 2 2024-04-01T13:30:33.114 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.114 INFO:tasks.rados.rados.0.smithi059.stdout:13: finishing write tid 3 to smithi05931183-13 2024-04-01T13:30:33.114 INFO:tasks.rados.rados.0.smithi059.stdout:13: finishing write tid 4 to smithi05931183-13 2024-04-01T13:30:33.114 INFO:tasks.rados.rados.0.smithi059.stdout:13: oid 13 updating version 0 to 6 2024-04-01T13:30:33.114 INFO:tasks.rados.rados.0.smithi059.stdout:13: oid 13 version 6 is already newer than 5 2024-04-01T13:30:33.114 INFO:tasks.rados.rados.0.smithi059.stdout:13: oid 13 version 6 is already newer than 4 2024-04-01T13:30:33.114 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-01T13:30:33.114 INFO:tasks.rados.rados.0.smithi059.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-04-01T13:30:33.115 INFO:tasks.rados.rados.0.smithi059.stdout:13: done (15 left) 2024-04-01T13:30:33.115 INFO:tasks.rados.rados.0.smithi059.stdout:35: write initial oid 35 2024-04-01T13:30:33.115 INFO:tasks.rados.rados.0.smithi059.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-04-01T13:30:33.123 INFO:tasks.rados.rados.0.smithi059.stdout:35: writing smithi05931183-35 from 571438 to 1195316 tid 1 2024-04-01T13:30:33.131 INFO:tasks.rados.rados.0.smithi059.stdout:35: writing smithi05931183-35 from 1806623 to 2436290 tid 2 2024-04-01T13:30:33.131 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.131 INFO:tasks.rados.rados.0.smithi059.stdout:30: finishing write tid 1 to smithi05931183-30 2024-04-01T13:30:33.131 INFO:tasks.rados.rados.0.smithi059.stdout:27: finishing write tid 2 to smithi05931183-27 2024-04-01T13:30:33.131 INFO:tasks.rados.rados.0.smithi059.stdout:22: finishing write tid 3 to smithi05931183-22 2024-04-01T13:30:33.131 INFO:tasks.rados.rados.0.smithi059.stdout:22: finishing write tid 4 to smithi05931183-22 2024-04-01T13:30:33.132 INFO:tasks.rados.rados.0.smithi059.stdout:22: oid 22 updating version 0 to 1 2024-04-01T13:30:33.132 INFO:tasks.rados.rados.0.smithi059.stdout:22: oid 22 updating version 1 to 3 2024-04-01T13:30:33.132 INFO:tasks.rados.rados.0.smithi059.stdout:22: oid 22 version 3 is already newer than 2 2024-04-01T13:30:33.132 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-01T13:30:33.132 INFO:tasks.rados.rados.0.smithi059.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-04-01T13:30:33.132 INFO:tasks.rados.rados.0.smithi059.stdout:22: done (15 left) 2024-04-01T13:30:33.132 INFO:tasks.rados.rados.0.smithi059.stdout:36: write initial oid 36 2024-04-01T13:30:33.132 INFO:tasks.rados.rados.0.smithi059.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-04-01T13:30:33.138 INFO:tasks.rados.rados.0.smithi059.stdout:36: writing smithi05931183-36 from 588245 to 1287372 tid 1 2024-04-01T13:30:33.145 INFO:tasks.rados.rados.0.smithi059.stdout:36: writing smithi05931183-36 from 1865105 to 2354783 tid 2 2024-04-01T13:30:33.145 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.146 INFO:tasks.rados.rados.0.smithi059.stdout:20: finishing write tid 1 to smithi05931183-20 2024-04-01T13:30:33.146 INFO:tasks.rados.rados.0.smithi059.stdout:21: finishing write tid 2 to smithi05931183-21 2024-04-01T13:30:33.153 INFO:tasks.rados.rados.0.smithi059.stdout:26: finishing write tid 1 to smithi05931183-26 2024-04-01T13:30:33.169 INFO:tasks.rados.rados.0.smithi059.stdout:30: finishing write tid 2 to smithi05931183-30 2024-04-01T13:30:33.178 INFO:tasks.rados.rados.0.smithi059.stdout:19: finishing write tid 1 to smithi05931183-19 2024-04-01T13:30:33.182 INFO:tasks.rados.rados.0.smithi059.stdout:27: finishing write tid 3 to smithi05931183-27 2024-04-01T13:30:33.191 INFO:tasks.rados.rados.0.smithi059.stdout:27: finishing write tid 4 to smithi05931183-27 2024-04-01T13:30:33.191 INFO:tasks.rados.rados.0.smithi059.stdout:27: oid 27 updating version 0 to 6 2024-04-01T13:30:33.191 INFO:tasks.rados.rados.0.smithi059.stdout:27: oid 27 version 6 is already newer than 4 2024-04-01T13:30:33.191 INFO:tasks.rados.rados.0.smithi059.stdout:27: oid 27 version 6 is already newer than 5 2024-04-01T13:30:33.191 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-01T13:30:33.191 INFO:tasks.rados.rados.0.smithi059.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-04-01T13:30:33.191 INFO:tasks.rados.rados.0.smithi059.stdout:27: done (15 left) 2024-04-01T13:30:33.191 INFO:tasks.rados.rados.0.smithi059.stdout:37: write initial oid 37 2024-04-01T13:30:33.191 INFO:tasks.rados.rados.0.smithi059.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-04-01T13:30:33.198 INFO:tasks.rados.rados.0.smithi059.stdout:37: writing smithi05931183-37 from 605052 to 1379428 tid 1 2024-04-01T13:30:33.207 INFO:tasks.rados.rados.0.smithi059.stdout:37: writing smithi05931183-37 from 1923587 to 2556923 tid 2 2024-04-01T13:30:33.207 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.207 INFO:tasks.rados.rados.0.smithi059.stdout:20: finishing write tid 2 to smithi05931183-20 2024-04-01T13:30:33.207 INFO:tasks.rados.rados.0.smithi059.stdout:21: finishing write tid 3 to smithi05931183-21 2024-04-01T13:30:33.209 INFO:tasks.rados.rados.0.smithi059.stdout:36: finishing write tid 1 to smithi05931183-36 2024-04-01T13:30:33.213 INFO:tasks.rados.rados.0.smithi059.stdout:20: finishing write tid 3 to smithi05931183-20 2024-04-01T13:30:33.213 INFO:tasks.rados.rados.0.smithi059.stdout:20: finishing write tid 4 to smithi05931183-20 2024-04-01T13:30:33.213 INFO:tasks.rados.rados.0.smithi059.stdout:20: oid 20 updating version 0 to 9 2024-04-01T13:30:33.213 INFO:tasks.rados.rados.0.smithi059.stdout:20: oid 20 version 9 is already newer than 9 2024-04-01T13:30:33.213 INFO:tasks.rados.rados.0.smithi059.stdout:20: oid 20 version 9 is already newer than 7 2024-04-01T13:30:33.213 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 20 v 9 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-01T13:30:33.213 INFO:tasks.rados.rados.0.smithi059.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-04-01T13:30:33.213 INFO:tasks.rados.rados.0.smithi059.stdout:20: done (15 left) 2024-04-01T13:30:33.213 INFO:tasks.rados.rados.0.smithi059.stdout:38: write initial oid 38 2024-04-01T13:30:33.213 INFO:tasks.rados.rados.0.smithi059.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-04-01T13:30:33.219 INFO:tasks.rados.rados.0.smithi059.stdout:38: writing smithi05931183-38 from 621859 to 1071484 tid 1 2024-04-01T13:30:33.229 INFO:tasks.rados.rados.0.smithi059.stdout:38: writing smithi05931183-38 from 1865716 to 2621859 tid 2 2024-04-01T13:30:33.229 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.229 INFO:tasks.rados.rados.0.smithi059.stdout:21: finishing write tid 4 to smithi05931183-21 2024-04-01T13:30:33.229 INFO:tasks.rados.rados.0.smithi059.stdout:21: oid 21 updating version 0 to 3 2024-04-01T13:30:33.229 INFO:tasks.rados.rados.0.smithi059.stdout:21: oid 21 version 3 is already newer than 2 2024-04-01T13:30:33.229 INFO:tasks.rados.rados.0.smithi059.stdout:21: oid 21 version 3 is already newer than 1 2024-04-01T13:30:33.229 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 21 v 3 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-01T13:30:33.229 INFO:tasks.rados.rados.0.smithi059.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-04-01T13:30:33.229 INFO:tasks.rados.rados.0.smithi059.stdout:21: done (15 left) 2024-04-01T13:30:33.229 INFO:tasks.rados.rados.0.smithi059.stdout:39: write initial oid 39 2024-04-01T13:30:33.229 INFO:tasks.rados.rados.0.smithi059.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-04-01T13:30:33.235 INFO:tasks.rados.rados.0.smithi059.stdout:39: writing smithi05931183-39 from 638666 to 1163540 tid 1 2024-04-01T13:30:33.247 INFO:tasks.rados.rados.0.smithi059.stdout:39: writing smithi05931183-39 from 1924198 to 2561203 tid 2 2024-04-01T13:30:33.247 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.247 INFO:tasks.rados.rados.0.smithi059.stdout:26: finishing write tid 2 to smithi05931183-26 2024-04-01T13:30:33.247 INFO:tasks.rados.rados.0.smithi059.stdout:30: finishing write tid 3 to smithi05931183-30 2024-04-01T13:30:33.247 INFO:tasks.rados.rados.0.smithi059.stdout:30: finishing write tid 4 to smithi05931183-30 2024-04-01T13:30:33.247 INFO:tasks.rados.rados.0.smithi059.stdout:30: oid 30 updating version 0 to 9 2024-04-01T13:30:33.247 INFO:tasks.rados.rados.0.smithi059.stdout:30: oid 30 version 9 is already newer than 7 2024-04-01T13:30:33.247 INFO:tasks.rados.rados.0.smithi059.stdout:30: oid 30 version 9 is already newer than 8 2024-04-01T13:30:33.247 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 30 v 9 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-01T13:30:33.247 INFO:tasks.rados.rados.0.smithi059.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-04-01T13:30:33.248 INFO:tasks.rados.rados.0.smithi059.stdout:30: done (15 left) 2024-04-01T13:30:33.248 INFO:tasks.rados.rados.0.smithi059.stdout:40: write initial oid 40 2024-04-01T13:30:33.248 INFO:tasks.rados.rados.0.smithi059.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-04-01T13:30:33.253 INFO:tasks.rados.rados.0.smithi059.stdout:40: writing smithi05931183-40 from 655473 to 1371949 tid 1 2024-04-01T13:30:33.261 INFO:tasks.rados.rados.0.smithi059.stdout:40: writing smithi05931183-40 from 2099033 to 2655473 tid 2 2024-04-01T13:30:33.262 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.262 INFO:tasks.rados.rados.0.smithi059.stdout:19: finishing write tid 2 to smithi05931183-19 2024-04-01T13:30:33.262 INFO:tasks.rados.rados.0.smithi059.stdout:19: finishing write tid 3 to smithi05931183-19 2024-04-01T13:30:33.265 INFO:tasks.rados.rados.0.smithi059.stdout:33: finishing write tid 1 to smithi05931183-33 2024-04-01T13:30:33.269 INFO:tasks.rados.rados.0.smithi059.stdout:19: finishing write tid 4 to smithi05931183-19 2024-04-01T13:30:33.270 INFO:tasks.rados.rados.0.smithi059.stdout:19: oid 19 updating version 0 to 9 2024-04-01T13:30:33.270 INFO:tasks.rados.rados.0.smithi059.stdout:19: oid 19 version 9 is already newer than 9 2024-04-01T13:30:33.270 INFO:tasks.rados.rados.0.smithi059.stdout:19: oid 19 version 9 is already newer than 7 2024-04-01T13:30:33.270 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 19 v 9 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-01T13:30:33.270 INFO:tasks.rados.rados.0.smithi059.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-04-01T13:30:33.270 INFO:tasks.rados.rados.0.smithi059.stdout:19: done (15 left) 2024-04-01T13:30:33.270 INFO:tasks.rados.rados.0.smithi059.stdout:41: write initial oid 41 2024-04-01T13:30:33.270 INFO:tasks.rados.rados.0.smithi059.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-04-01T13:30:33.280 INFO:tasks.rados.rados.0.smithi059.stdout:41: writing smithi05931183-41 from 672280 to 1464005 tid 1 2024-04-01T13:30:33.288 INFO:tasks.rados.rados.0.smithi059.stdout:41: writing smithi05931183-41 from 2157515 to 2672280 tid 2 2024-04-01T13:30:33.288 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.288 INFO:tasks.rados.rados.0.smithi059.stdout:23: finishing write tid 1 to smithi05931183-23 2024-04-01T13:30:33.288 INFO:tasks.rados.rados.0.smithi059.stdout:38: finishing write tid 1 to smithi05931183-38 2024-04-01T13:30:33.288 INFO:tasks.rados.rados.0.smithi059.stdout:36: finishing write tid 2 to smithi05931183-36 2024-04-01T13:30:33.292 INFO:tasks.rados.rados.0.smithi059.stdout:37: finishing write tid 1 to smithi05931183-37 2024-04-01T13:30:33.310 INFO:tasks.rados.rados.0.smithi059.stdout:31: finishing write tid 1 to smithi05931183-31 2024-04-01T13:30:33.315 INFO:tasks.rados.rados.0.smithi059.stdout:28: finishing write tid 1 to smithi05931183-28 2024-04-01T13:30:33.322 INFO:tasks.rados.rados.0.smithi059.stdout:38: finishing write tid 2 to smithi05931183-38 2024-04-01T13:30:33.322 INFO:tasks.rados.rados.0.smithi059.stdout:26: finishing write tid 3 to smithi05931183-26 2024-04-01T13:30:33.324 INFO:tasks.rados.rados.0.smithi059.stdout:26: finishing write tid 4 to smithi05931183-26 2024-04-01T13:30:33.324 INFO:tasks.rados.rados.0.smithi059.stdout:26: oid 26 updating version 0 to 6 2024-04-01T13:30:33.324 INFO:tasks.rados.rados.0.smithi059.stdout:26: oid 26 version 6 is already newer than 5 2024-04-01T13:30:33.324 INFO:tasks.rados.rados.0.smithi059.stdout:26: oid 26 version 6 is already newer than 4 2024-04-01T13:30:33.324 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-01T13:30:33.324 INFO:tasks.rados.rados.0.smithi059.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-04-01T13:30:33.324 INFO:tasks.rados.rados.0.smithi059.stdout:26: done (15 left) 2024-04-01T13:30:33.324 INFO:tasks.rados.rados.0.smithi059.stdout:42: write initial oid 42 2024-04-01T13:30:33.324 INFO:tasks.rados.rados.0.smithi059.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-04-01T13:30:33.330 INFO:tasks.rados.rados.0.smithi059.stdout:42: writing smithi05931183-42 from 689087 to 1156061 tid 1 2024-04-01T13:30:33.342 INFO:tasks.rados.rados.0.smithi059.stdout:42: writing smithi05931183-42 from 1699644 to 2483976 tid 2 2024-04-01T13:30:33.342 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.342 INFO:tasks.rados.rados.0.smithi059.stdout:36: finishing write tid 3 to smithi05931183-36 2024-04-01T13:30:33.346 INFO:tasks.rados.rados.0.smithi059.stdout:36: finishing write tid 4 to smithi05931183-36 2024-04-01T13:30:33.347 INFO:tasks.rados.rados.0.smithi059.stdout:36: oid 36 updating version 0 to 9 2024-04-01T13:30:33.347 INFO:tasks.rados.rados.0.smithi059.stdout:36: oid 36 version 9 is already newer than 7 2024-04-01T13:30:33.347 INFO:tasks.rados.rados.0.smithi059.stdout:36: oid 36 version 9 is already newer than 8 2024-04-01T13:30:33.347 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-01T13:30:33.347 INFO:tasks.rados.rados.0.smithi059.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-04-01T13:30:33.347 INFO:tasks.rados.rados.0.smithi059.stdout:36: done (15 left) 2024-04-01T13:30:33.347 INFO:tasks.rados.rados.0.smithi059.stdout:43: write initial oid 43 2024-04-01T13:30:33.347 INFO:tasks.rados.rados.0.smithi059.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-04-01T13:30:33.354 INFO:tasks.rados.rados.0.smithi059.stdout:43: writing smithi05931183-43 from 705894 to 1248117 tid 1 2024-04-01T13:30:33.361 INFO:tasks.rados.rados.0.smithi059.stdout:43: writing smithi05931183-43 from 1758126 to 2402469 tid 2 2024-04-01T13:30:33.361 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.361 INFO:tasks.rados.rados.0.smithi059.stdout:41: finishing write tid 1 to smithi05931183-41 2024-04-01T13:30:33.362 INFO:tasks.rados.rados.0.smithi059.stdout:38: finishing write tid 3 to smithi05931183-38 2024-04-01T13:30:33.362 INFO:tasks.rados.rados.0.smithi059.stdout:38: finishing write tid 4 to smithi05931183-38 2024-04-01T13:30:33.362 INFO:tasks.rados.rados.0.smithi059.stdout:38: oid 38 updating version 0 to 6 2024-04-01T13:30:33.362 INFO:tasks.rados.rados.0.smithi059.stdout:38: oid 38 version 6 is already newer than 5 2024-04-01T13:30:33.362 INFO:tasks.rados.rados.0.smithi059.stdout:38: oid 38 version 6 is already newer than 4 2024-04-01T13:30:33.362 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 38 v 6 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-01T13:30:33.362 INFO:tasks.rados.rados.0.smithi059.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-04-01T13:30:33.362 INFO:tasks.rados.rados.0.smithi059.stdout:38: done (15 left) 2024-04-01T13:30:33.362 INFO:tasks.rados.rados.0.smithi059.stdout:44: write initial oid 44 2024-04-01T13:30:33.362 INFO:tasks.rados.rados.0.smithi059.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-04-01T13:30:33.370 INFO:tasks.rados.rados.0.smithi059.stdout:44: writing smithi05931183-44 from 722701 to 1340173 tid 1 2024-04-01T13:30:33.382 INFO:tasks.rados.rados.0.smithi059.stdout:44: writing smithi05931183-44 from 1816608 to 2604609 tid 2 2024-04-01T13:30:33.382 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.382 INFO:tasks.rados.rados.0.smithi059.stdout:25: finishing write tid 1 to smithi05931183-25 2024-04-01T13:30:33.382 INFO:tasks.rados.rados.0.smithi059.stdout:33: finishing write tid 2 to smithi05931183-33 2024-04-01T13:30:33.382 INFO:tasks.rados.rados.0.smithi059.stdout:33: finishing write tid 3 to smithi05931183-33 2024-04-01T13:30:33.382 INFO:tasks.rados.rados.0.smithi059.stdout:33: finishing write tid 4 to smithi05931183-33 2024-04-01T13:30:33.382 INFO:tasks.rados.rados.0.smithi059.stdout:33: oid 33 updating version 0 to 7 2024-04-01T13:30:33.382 INFO:tasks.rados.rados.0.smithi059.stdout:33: oid 33 updating version 7 to 9 2024-04-01T13:30:33.382 INFO:tasks.rados.rados.0.smithi059.stdout:33: oid 33 version 9 is already newer than 9 2024-04-01T13:30:33.382 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 33 v 9 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-01T13:30:33.382 INFO:tasks.rados.rados.0.smithi059.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-04-01T13:30:33.382 INFO:tasks.rados.rados.0.smithi059.stdout:33: done (15 left) 2024-04-01T13:30:33.382 INFO:tasks.rados.rados.0.smithi059.stdout:45: write initial oid 45 2024-04-01T13:30:33.383 INFO:tasks.rados.rados.0.smithi059.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-04-01T13:30:33.392 INFO:tasks.rados.rados.0.smithi059.stdout:45: writing smithi05931183-45 from 739508 to 1432229 tid 1 2024-04-01T13:30:33.399 INFO:tasks.rados.rados.0.smithi059.stdout:45: writing smithi05931183-45 from 1875090 to 2523102 tid 2 2024-04-01T13:30:33.399 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.399 INFO:tasks.rados.rados.0.smithi059.stdout:31: finishing write tid 2 to smithi05931183-31 2024-04-01T13:30:33.399 INFO:tasks.rados.rados.0.smithi059.stdout:23: finishing write tid 2 to smithi05931183-23 2024-04-01T13:30:33.399 INFO:tasks.rados.rados.0.smithi059.stdout:37: finishing write tid 2 to smithi05931183-37 2024-04-01T13:30:33.399 INFO:tasks.rados.rados.0.smithi059.stdout:23: finishing write tid 3 to smithi05931183-23 2024-04-01T13:30:33.400 INFO:tasks.rados.rados.0.smithi059.stdout:31: finishing write tid 3 to smithi05931183-31 2024-04-01T13:30:33.403 INFO:tasks.rados.rados.0.smithi059.stdout:31: finishing write tid 4 to smithi05931183-31 2024-04-01T13:30:33.403 INFO:tasks.rados.rados.0.smithi059.stdout:31: oid 31 updating version 0 to 10 2024-04-01T13:30:33.403 INFO:tasks.rados.rados.0.smithi059.stdout:31: oid 31 updating version 10 to 12 2024-04-01T13:30:33.403 INFO:tasks.rados.rados.0.smithi059.stdout:31: oid 31 version 12 is already newer than 12 2024-04-01T13:30:33.403 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 31 v 12 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-01T13:30:33.403 INFO:tasks.rados.rados.0.smithi059.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-04-01T13:30:33.403 INFO:tasks.rados.rados.0.smithi059.stdout:31: done (15 left) 2024-04-01T13:30:33.403 INFO:tasks.rados.rados.0.smithi059.stdout:46: write initial oid 46 2024-04-01T13:30:33.403 INFO:tasks.rados.rados.0.smithi059.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-04-01T13:30:33.413 INFO:tasks.rados.rados.0.smithi059.stdout:46: writing smithi05931183-46 from 756315 to 1524285 tid 1 2024-04-01T13:30:33.423 INFO:tasks.rados.rados.0.smithi059.stdout:46: writing smithi05931183-46 from 1933572 to 2725242 tid 2 2024-04-01T13:30:33.423 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.424 INFO:tasks.rados.rados.0.smithi059.stdout:23: finishing write tid 4 to smithi05931183-23 2024-04-01T13:30:33.424 INFO:tasks.rados.rados.0.smithi059.stdout:23: oid 23 updating version 0 to 12 2024-04-01T13:30:33.424 INFO:tasks.rados.rados.0.smithi059.stdout:23: oid 23 version 12 is already newer than 12 2024-04-01T13:30:33.424 INFO:tasks.rados.rados.0.smithi059.stdout:23: oid 23 version 12 is already newer than 10 2024-04-01T13:30:33.424 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 23 v 12 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-01T13:30:33.424 INFO:tasks.rados.rados.0.smithi059.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-04-01T13:30:33.424 INFO:tasks.rados.rados.0.smithi059.stdout:23: done (15 left) 2024-04-01T13:30:33.424 INFO:tasks.rados.rados.0.smithi059.stdout:47: write initial oid 47 2024-04-01T13:30:33.424 INFO:tasks.rados.rados.0.smithi059.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-04-01T13:30:33.432 INFO:tasks.rados.rados.0.smithi059.stdout:47: writing smithi05931183-47 from 773122 to 1332694 tid 1 2024-04-01T13:30:33.440 INFO:tasks.rados.rados.0.smithi059.stdout:47: writing smithi05931183-47 from 1992054 to 2643735 tid 2 2024-04-01T13:30:33.440 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.440 INFO:tasks.rados.rados.0.smithi059.stdout:41: finishing write tid 2 to smithi05931183-41 2024-04-01T13:30:33.440 INFO:tasks.rados.rados.0.smithi059.stdout:28: finishing write tid 2 to smithi05931183-28 2024-04-01T13:30:33.440 INFO:tasks.rados.rados.0.smithi059.stdout:45: finishing write tid 1 to smithi05931183-45 2024-04-01T13:30:33.440 INFO:tasks.rados.rados.0.smithi059.stdout:28: finishing write tid 3 to smithi05931183-28 2024-04-01T13:30:33.445 INFO:tasks.rados.rados.0.smithi059.stdout:32: finishing write tid 1 to smithi05931183-32 2024-04-01T13:30:33.447 INFO:tasks.rados.rados.0.smithi059.stdout:28: finishing write tid 4 to smithi05931183-28 2024-04-01T13:30:33.448 INFO:tasks.rados.rados.0.smithi059.stdout:28: oid 28 updating version 0 to 6 2024-04-01T13:30:33.448 INFO:tasks.rados.rados.0.smithi059.stdout:28: oid 28 version 6 is already newer than 4 2024-04-01T13:30:33.448 INFO:tasks.rados.rados.0.smithi059.stdout:28: oid 28 version 6 is already newer than 6 2024-04-01T13:30:33.448 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-01T13:30:33.448 INFO:tasks.rados.rados.0.smithi059.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-04-01T13:30:33.448 INFO:tasks.rados.rados.0.smithi059.stdout:28: done (15 left) 2024-04-01T13:30:33.448 INFO:tasks.rados.rados.0.smithi059.stdout:48: write initial oid 48 2024-04-01T13:30:33.448 INFO:tasks.rados.rados.0.smithi059.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-04-01T13:30:33.458 INFO:tasks.rados.rados.0.smithi059.stdout:48: writing smithi05931183-48 from 789929 to 1424750 tid 1 2024-04-01T13:30:33.461 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T13:30:33.470 INFO:tasks.rados.rados.0.smithi059.stdout:48: writing smithi05931183-48 from 2050536 to 2789929 tid 2 2024-04-01T13:30:33.470 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.473 INFO:tasks.rados.rados.0.smithi059.stdout:41: finishing write tid 3 to smithi05931183-41 2024-04-01T13:30:33.474 INFO:tasks.rados.rados.0.smithi059.stdout:41: finishing write tid 4 to smithi05931183-41 2024-04-01T13:30:33.474 INFO:tasks.rados.rados.0.smithi059.stdout:41: oid 41 updating version 0 to 4 2024-04-01T13:30:33.475 INFO:tasks.rados.rados.0.smithi059.stdout:41: oid 41 updating version 4 to 6 2024-04-01T13:30:33.475 INFO:tasks.rados.rados.0.smithi059.stdout:41: oid 41 version 6 is already newer than 5 2024-04-01T13:30:33.475 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 41 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-01T13:30:33.475 INFO:tasks.rados.rados.0.smithi059.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-04-01T13:30:33.475 INFO:tasks.rados.rados.0.smithi059.stdout:41: done (15 left) 2024-04-01T13:30:33.475 INFO:tasks.rados.rados.0.smithi059.stdout:49: write initial oid 49 2024-04-01T13:30:33.475 INFO:tasks.rados.rados.0.smithi059.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-04-01T13:30:33.481 INFO:tasks.rados.rados.0.smithi059.stdout:49: writing smithi05931183-49 from 406736 to 1116806 tid 1 2024-04-01T13:30:33.491 INFO:tasks.rados.rados.0.smithi059.stdout:49: writing smithi05931183-49 from 1709018 to 2364368 tid 2 2024-04-01T13:30:33.491 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.491 INFO:tasks.rados.rados.0.smithi059.stdout:46: finishing write tid 1 to smithi05931183-46 2024-04-01T13:30:33.491 INFO:tasks.rados.rados.0.smithi059.stdout:45: finishing write tid 2 to smithi05931183-45 2024-04-01T13:30:33.491 INFO:tasks.rados.rados.0.smithi059.stdout:25: finishing write tid 2 to smithi05931183-25 2024-04-01T13:30:33.491 INFO:tasks.rados.rados.0.smithi059.stdout:47: finishing write tid 1 to smithi05931183-47 2024-04-01T13:30:33.493 INFO:tasks.rados.rados.0.smithi059.stdout:37: finishing write tid 3 to smithi05931183-37 2024-04-01T13:30:33.494 INFO:tasks.rados.rados.0.smithi059.stdout:25: finishing write tid 3 to smithi05931183-25 2024-04-01T13:30:33.498 INFO:tasks.rados.rados.0.smithi059.stdout:35: finishing write tid 1 to smithi05931183-35 2024-04-01T13:30:33.498 INFO:tasks.rados.rados.0.smithi059.stdout:34: finishing write tid 1 to smithi05931183-34 2024-04-01T13:30:33.500 INFO:tasks.rados.rados.0.smithi059.stdout:37: finishing write tid 4 to smithi05931183-37 2024-04-01T13:30:33.500 INFO:tasks.rados.rados.0.smithi059.stdout:37: oid 37 updating version 0 to 7 2024-04-01T13:30:33.500 INFO:tasks.rados.rados.0.smithi059.stdout:37: oid 37 updating version 7 to 8 2024-04-01T13:30:33.500 INFO:tasks.rados.rados.0.smithi059.stdout:37: oid 37 updating version 8 to 9 2024-04-01T13:30:33.500 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 37 v 9 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-01T13:30:33.500 INFO:tasks.rados.rados.0.smithi059.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-04-01T13:30:33.501 INFO:tasks.rados.rados.0.smithi059.stdout:37: done (15 left) 2024-04-01T13:30:33.501 INFO:tasks.rados.rados.0.smithi059.stdout:50: write initial oid 50 2024-04-01T13:30:33.501 INFO:tasks.rados.rados.0.smithi059.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-04-01T13:30:33.510 INFO:tasks.rados.rados.0.smithi059.stdout:50: writing smithi05931183-50 from 423543 to 1208862 tid 1 2024-04-01T13:30:33.519 INFO:tasks.rados.rados.0.smithi059.stdout:50: writing smithi05931183-50 from 1767500 to 2566508 tid 2 2024-04-01T13:30:33.519 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.519 INFO:tasks.rados.rados.0.smithi059.stdout:25: finishing write tid 4 to smithi05931183-25 2024-04-01T13:30:33.519 INFO:tasks.rados.rados.0.smithi059.stdout:25: oid 25 updating version 0 to 10 2024-04-01T13:30:33.519 INFO:tasks.rados.rados.0.smithi059.stdout:25: oid 25 updating version 10 to 12 2024-04-01T13:30:33.519 INFO:tasks.rados.rados.0.smithi059.stdout:25: oid 25 version 12 is already newer than 12 2024-04-01T13:30:33.519 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 25 v 12 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-01T13:30:33.519 INFO:tasks.rados.rados.0.smithi059.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-04-01T13:30:33.519 INFO:tasks.rados.rados.0.smithi059.stdout:25: done (15 left) 2024-04-01T13:30:33.519 INFO:tasks.rados.rados.0.smithi059.stdout:51: write (excl) oid 2 current snap is 0 2024-04-01T13:30:33.519 INFO:tasks.rados.rados.0.smithi059.stdout:51: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2024-04-01T13:30:33.525 INFO:tasks.rados.rados.0.smithi059.stdout:51: writing smithi05931183-2 from 440350 to 900918 tid 1 2024-04-01T13:30:33.531 INFO:tasks.rados.rados.0.smithi059.stdout:51: writing smithi05931183-2 from 1309629 to 1852295 tid 2 2024-04-01T13:30:33.539 INFO:tasks.rados.rados.0.smithi059.stdout:51: writing smithi05931183-2 from 2323973 to 2840350 tid 3 2024-04-01T13:30:33.539 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.539 INFO:tasks.rados.rados.0.smithi059.stdout:29: finishing write tid 1 to smithi05931183-29 2024-04-01T13:30:33.539 INFO:tasks.rados.rados.0.smithi059.stdout:45: finishing write tid 3 to smithi05931183-45 2024-04-01T13:30:33.539 INFO:tasks.rados.rados.0.smithi059.stdout:45: finishing write tid 4 to smithi05931183-45 2024-04-01T13:30:33.539 INFO:tasks.rados.rados.0.smithi059.stdout:45: oid 45 updating version 0 to 9 2024-04-01T13:30:33.539 INFO:tasks.rados.rados.0.smithi059.stdout:45: oid 45 version 9 is already newer than 8 2024-04-01T13:30:33.539 INFO:tasks.rados.rados.0.smithi059.stdout:45: oid 45 version 9 is already newer than 7 2024-04-01T13:30:33.539 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 45 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-01T13:30:33.539 INFO:tasks.rados.rados.0.smithi059.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-04-01T13:30:33.540 INFO:tasks.rados.rados.0.smithi059.stdout:45: done (15 left) 2024-04-01T13:30:33.540 INFO:tasks.rados.rados.0.smithi059.stdout:52: write (excl) oid 14 current snap is 0 2024-04-01T13:30:33.540 INFO:tasks.rados.rados.0.smithi059.stdout:52: seq_num 51 ranges {457157=535817,1768111=402677} 2024-04-01T13:30:33.544 INFO:tasks.rados.rados.0.smithi059.stdout:52: writing smithi05931183-14 from 457157 to 992974 tid 1 2024-04-01T13:30:33.552 INFO:tasks.rados.rados.0.smithi059.stdout:52: writing smithi05931183-14 from 1768111 to 2170788 tid 2 2024-04-01T13:30:33.552 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.552 INFO:tasks.rados.rados.0.smithi059.stdout:46: finishing write tid 2 to smithi05931183-46 2024-04-01T13:30:33.562 INFO:tasks.rados.rados.0.smithi059.stdout:46: finishing write tid 3 to smithi05931183-46 2024-04-01T13:30:33.563 INFO:tasks.rados.rados.0.smithi059.stdout:32: finishing write tid 2 to smithi05931183-32 2024-04-01T13:30:33.564 INFO:tasks.rados.rados.0.smithi059.stdout:46: finishing write tid 4 to smithi05931183-46 2024-04-01T13:30:33.564 INFO:tasks.rados.rados.0.smithi059.stdout:46: oid 46 updating version 0 to 3 2024-04-01T13:30:33.564 INFO:tasks.rados.rados.0.smithi059.stdout:46: oid 46 version 3 is already newer than 1 2024-04-01T13:30:33.564 INFO:tasks.rados.rados.0.smithi059.stdout:46: oid 46 version 3 is already newer than 3 2024-04-01T13:30:33.564 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 46 v 3 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-01T13:30:33.564 INFO:tasks.rados.rados.0.smithi059.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-04-01T13:30:33.564 INFO:tasks.rados.rados.0.smithi059.stdout:46: done (15 left) 2024-04-01T13:30:33.564 INFO:tasks.rados.rados.0.smithi059.stdout:53: read oid 20 snap -1 2024-04-01T13:30:33.564 INFO:tasks.rados.rados.0.smithi059.stdout:53: expect (ObjNum 19 snap 0 seq_num 19) 2024-04-01T13:30:33.565 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-01T13:30:33.566 INFO:tasks.rados.rados.0.smithi059.stdout:34: finishing write tid 2 to smithi05931183-34 2024-04-01T13:30:33.566 INFO:tasks.rados.rados.0.smithi059.stdout:43: finishing write tid 1 to smithi05931183-43 2024-04-01T13:30:33.574 INFO:tasks.rados.rados.0.smithi059.stdout:32: finishing write tid 3 to smithi05931183-32 2024-04-01T13:30:33.580 INFO:tasks.rados.rados.0.smithi059.stdout:32: finishing write tid 4 to smithi05931183-32 2024-04-01T13:30:33.580 INFO:tasks.rados.rados.0.smithi059.stdout:32: oid 32 updating version 0 to 7 2024-04-01T13:30:33.580 INFO:tasks.rados.rados.0.smithi059.stdout:32: oid 32 updating version 7 to 9 2024-04-01T13:30:33.580 INFO:tasks.rados.rados.0.smithi059.stdout:32: oid 32 version 9 is already newer than 9 2024-04-01T13:30:33.580 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 32 v 9 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-01T13:30:33.580 INFO:tasks.rados.rados.0.smithi059.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-04-01T13:30:33.580 INFO:tasks.rados.rados.0.smithi059.stdout:32: done (15 left) 2024-04-01T13:30:33.580 INFO:tasks.rados.rados.0.smithi059.stdout:54: delete oid 23 current snap is 0 2024-04-01T13:30:33.585 INFO:tasks.rados.rados.0.smithi059.stdout:47: finishing write tid 2 to smithi05931183-47 2024-04-01T13:30:33.593 INFO:tasks.rados.rados.0.smithi059.stdout:34: finishing write tid 3 to smithi05931183-34 2024-04-01T13:30:33.601 INFO:tasks.rados.rados.0.smithi059.stdout:51: finishing write tid 1 to smithi05931183-2 2024-04-01T13:30:33.601 INFO:tasks.rados.rados.0.smithi059.stdout:34: finishing write tid 4 to smithi05931183-34 2024-04-01T13:30:33.601 INFO:tasks.rados.rados.0.smithi059.stdout:34: oid 34 updating version 0 to 13 2024-04-01T13:30:33.601 INFO:tasks.rados.rados.0.smithi059.stdout:34: oid 34 updating version 13 to 15 2024-04-01T13:30:33.601 INFO:tasks.rados.rados.0.smithi059.stdout:34: oid 34 version 15 is already newer than 15 2024-04-01T13:30:33.601 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 34 v 15 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-01T13:30:33.601 INFO:tasks.rados.rados.0.smithi059.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-04-01T13:30:33.604 INFO:tasks.rados.rados.0.smithi059.stdout:29: finishing write tid 2 to smithi05931183-29 2024-04-01T13:30:33.609 INFO:tasks.rados.rados.0.smithi059.stdout:35: finishing write tid 2 to smithi05931183-35 2024-04-01T13:30:33.612 INFO:tasks.rados.rados.0.smithi059.stdout:35: finishing write tid 3 to smithi05931183-35 2024-04-01T13:30:33.614 INFO:tasks.rados.rados.0.smithi059.stdout:35: finishing write tid 4 to smithi05931183-35 2024-04-01T13:30:33.614 INFO:tasks.rados.rados.0.smithi059.stdout:35: oid 35 updating version 0 to 12 2024-04-01T13:30:33.615 INFO:tasks.rados.rados.0.smithi059.stdout:35: oid 35 version 12 is already newer than 12 2024-04-01T13:30:33.615 INFO:tasks.rados.rados.0.smithi059.stdout:35: oid 35 version 12 is already newer than 10 2024-04-01T13:30:33.615 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 35 v 12 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-01T13:30:33.615 INFO:tasks.rados.rados.0.smithi059.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-04-01T13:30:33.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:33.631+0000 7fe936b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:33.626 INFO:tasks.rados.rados.0.smithi059.stdout:47: finishing write tid 3 to smithi05931183-47 2024-04-01T13:30:33.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:33.640+0000 7fe936b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:33.628 INFO:tasks.rados.rados.0.smithi059.stdout:47: finishing write tid 4 to smithi05931183-47 2024-04-01T13:30:33.628 INFO:tasks.rados.rados.0.smithi059.stdout:47: oid 47 updating version 0 to 9 2024-04-01T13:30:33.628 INFO:tasks.rados.rados.0.smithi059.stdout:47: oid 47 version 9 is already newer than 8 2024-04-01T13:30:33.628 INFO:tasks.rados.rados.0.smithi059.stdout:47: oid 47 version 9 is already newer than 7 2024-04-01T13:30:33.628 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 47 v 9 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-01T13:30:33.628 INFO:tasks.rados.rados.0.smithi059.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-04-01T13:30:33.636 INFO:tasks.rados.rados.0.smithi059.stdout:48: finishing write tid 1 to smithi05931183-48 2024-04-01T13:30:33.637 INFO:tasks.rados.rados.0.smithi059.stdout:51: finishing write tid 2 to smithi05931183-2 2024-04-01T13:30:33.645 INFO:tasks.rados.rados.0.smithi059.stdout:52: finishing write tid 1 to smithi05931183-14 2024-04-01T13:30:33.670 INFO:tasks.rados.rados.0.smithi059.stdout:51: finishing write tid 3 to smithi05931183-2 2024-04-01T13:30:33.677 INFO:tasks.rados.rados.0.smithi059.stdout:29: finishing write tid 3 to smithi05931183-29 2024-04-01T13:30:33.677 INFO:tasks.rados.rados.0.smithi059.stdout:50: finishing write tid 1 to smithi05931183-50 2024-04-01T13:30:33.677 INFO:tasks.rados.rados.0.smithi059.stdout:29: finishing write tid 4 to smithi05931183-29 2024-04-01T13:30:33.678 INFO:tasks.rados.rados.0.smithi059.stdout:29: oid 29 updating version 0 to 13 2024-04-01T13:30:33.678 INFO:tasks.rados.rados.0.smithi059.stdout:29: oid 29 updating version 13 to 15 2024-04-01T13:30:33.678 INFO:tasks.rados.rados.0.smithi059.stdout:29: oid 29 version 15 is already newer than 15 2024-04-01T13:30:33.678 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 29 v 15 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-01T13:30:33.678 INFO:tasks.rados.rados.0.smithi059.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-04-01T13:30:33.678 INFO:tasks.rados.rados.0.smithi059.stdout:51: finishing write tid 4 to smithi05931183-2 2024-04-01T13:30:33.678 INFO:tasks.rados.rados.0.smithi059.stdout:51: finishing write tid 5 to smithi05931183-2 2024-04-01T13:30:33.678 INFO:tasks.rados.rados.0.smithi059.stdout:51: oid 2 updating version 0 to 13 2024-04-01T13:30:33.678 INFO:tasks.rados.rados.0.smithi059.stdout:51: oid 2 version 13 is already newer than 10 2024-04-01T13:30:33.678 INFO:tasks.rados.rados.0.smithi059.stdout:51: oid 2 version 13 is already newer than 13 2024-04-01T13:30:33.679 INFO:tasks.rados.rados.0.smithi059.stdout:51: oid 2 version 13 is already newer than 11 2024-04-01T13:30:33.679 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 2 v 13 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-04-01T13:30:33.679 INFO:tasks.rados.rados.0.smithi059.stdout:51: left oid 2 (ObjNum 50 snap 0 seq_num 50) 2024-04-01T13:30:33.683 INFO:tasks.rados.rados.0.smithi059.stdout:39: finishing write tid 1 to smithi05931183-39 2024-04-01T13:30:33.694 INFO:tasks.rados.rados.0.smithi059.stdout:43: finishing write tid 2 to smithi05931183-43 2024-04-01T13:30:33.698 INFO:tasks.rados.rados.0.smithi059.stdout:43: finishing write tid 3 to smithi05931183-43 2024-04-01T13:30:33.702 INFO:tasks.rados.rados.0.smithi059.stdout:43: finishing write tid 4 to smithi05931183-43 2024-04-01T13:30:33.702 INFO:tasks.rados.rados.0.smithi059.stdout:43: oid 43 updating version 0 to 7 2024-04-01T13:30:33.702 INFO:tasks.rados.rados.0.smithi059.stdout:43: oid 43 updating version 7 to 9 2024-04-01T13:30:33.702 INFO:tasks.rados.rados.0.smithi059.stdout:43: oid 43 version 9 is already newer than 9 2024-04-01T13:30:33.702 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 43 v 9 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-01T13:30:33.702 INFO:tasks.rados.rados.0.smithi059.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-04-01T13:30:33.703 INFO:tasks.rados.rados.0.smithi059.stdout:40: finishing write tid 1 to smithi05931183-40 2024-04-01T13:30:33.705 INFO:tasks.rados.rados.0.smithi059.stdout:50: finishing write tid 2 to smithi05931183-50 2024-04-01T13:30:33.717 INFO:tasks.rados.rados.0.smithi059.stdout:39: finishing write tid 2 to smithi05931183-39 2024-04-01T13:30:33.720 INFO:tasks.rados.rados.0.smithi059.stdout:48: finishing write tid 2 to smithi05931183-48 2024-04-01T13:30:33.730 INFO:tasks.rados.rados.0.smithi059.stdout:44: finishing write tid 1 to smithi05931183-44 2024-04-01T13:30:33.734 INFO:tasks.rados.rados.0.smithi059.stdout:50: finishing write tid 3 to smithi05931183-50 2024-04-01T13:30:33.735 INFO:tasks.rados.rados.0.smithi059.stdout:52: finishing write tid 2 to smithi05931183-14 2024-04-01T13:30:33.746 INFO:tasks.rados.rados.0.smithi059.stdout:50: finishing write tid 4 to smithi05931183-50 2024-04-01T13:30:33.746 INFO:tasks.rados.rados.0.smithi059.stdout:50: oid 50 updating version 0 to 11 2024-04-01T13:30:33.746 INFO:tasks.rados.rados.0.smithi059.stdout:50: oid 50 version 11 is already newer than 10 2024-04-01T13:30:33.746 INFO:tasks.rados.rados.0.smithi059.stdout:50: oid 50 updating version 11 to 12 2024-04-01T13:30:33.746 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 50 v 12 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-01T13:30:33.746 INFO:tasks.rados.rados.0.smithi059.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-04-01T13:30:33.757 INFO:tasks.rados.rados.0.smithi059.stdout:40: finishing write tid 2 to smithi05931183-40 2024-04-01T13:30:33.760 INFO:tasks.rados.rados.0.smithi059.stdout:39: finishing write tid 3 to smithi05931183-39 2024-04-01T13:30:33.760 INFO:tasks.rados.rados.0.smithi059.stdout:48: finishing write tid 3 to smithi05931183-48 2024-04-01T13:30:33.764 INFO:tasks.rados.rados.0.smithi059.stdout:44: finishing write tid 2 to smithi05931183-44 2024-04-01T13:30:33.764 INFO:tasks.rados.rados.0.smithi059.stdout:48: finishing write tid 4 to smithi05931183-48 2024-04-01T13:30:33.764 INFO:tasks.rados.rados.0.smithi059.stdout:48: oid 48 updating version 0 to 15 2024-04-01T13:30:33.764 INFO:tasks.rados.rados.0.smithi059.stdout:48: oid 48 version 15 is already newer than 14 2024-04-01T13:30:33.764 INFO:tasks.rados.rados.0.smithi059.stdout:48: oid 48 version 15 is already newer than 13 2024-04-01T13:30:33.764 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 48 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-01T13:30:33.764 INFO:tasks.rados.rados.0.smithi059.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-04-01T13:30:33.764 INFO:tasks.rados.rados.0.smithi059.stdout:39: finishing write tid 4 to smithi05931183-39 2024-04-01T13:30:33.764 INFO:tasks.rados.rados.0.smithi059.stdout:39: oid 39 updating version 0 to 17 2024-04-01T13:30:33.764 INFO:tasks.rados.rados.0.smithi059.stdout:39: oid 39 updating version 17 to 18 2024-04-01T13:30:33.764 INFO:tasks.rados.rados.0.smithi059.stdout:39: oid 39 version 18 is already newer than 16 2024-04-01T13:30:33.764 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 39 v 18 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-01T13:30:33.764 INFO:tasks.rados.rados.0.smithi059.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-04-01T13:30:33.767 INFO:tasks.rados.rados.0.smithi059.stdout:52: finishing write tid 3 to smithi05931183-14 2024-04-01T13:30:33.769 INFO:tasks.rados.rados.0.smithi059.stdout:52: finishing write tid 4 to smithi05931183-14 2024-04-01T13:30:33.769 INFO:tasks.rados.rados.0.smithi059.stdout:52: oid 14 updating version 0 to 9 2024-04-01T13:30:33.770 INFO:tasks.rados.rados.0.smithi059.stdout:52: oid 14 version 9 is already newer than 7 2024-04-01T13:30:33.770 INFO:tasks.rados.rados.0.smithi059.stdout:52: oid 14 version 9 is already newer than 8 2024-04-01T13:30:33.770 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 14 v 9 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-04-01T13:30:33.770 INFO:tasks.rados.rados.0.smithi059.stdout:52: left oid 14 (ObjNum 51 snap 0 seq_num 51) 2024-04-01T13:30:33.770 INFO:tasks.rados.rados.0.smithi059.stdout:44: finishing write tid 3 to smithi05931183-44 2024-04-01T13:30:33.774 INFO:tasks.rados.rados.0.smithi059.stdout:44: finishing write tid 4 to smithi05931183-44 2024-04-01T13:30:33.774 INFO:tasks.rados.rados.0.smithi059.stdout:44: oid 44 updating version 0 to 16 2024-04-01T13:30:33.774 INFO:tasks.rados.rados.0.smithi059.stdout:44: oid 44 updating version 16 to 18 2024-04-01T13:30:33.774 INFO:tasks.rados.rados.0.smithi059.stdout:44: oid 44 version 18 is already newer than 18 2024-04-01T13:30:33.774 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 44 v 18 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-01T13:30:33.774 INFO:tasks.rados.rados.0.smithi059.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-04-01T13:30:33.778 INFO:tasks.rados.rados.0.smithi059.stdout:40: finishing write tid 3 to smithi05931183-40 2024-04-01T13:30:33.781 INFO:tasks.rados.rados.0.smithi059.stdout:40: finishing write tid 4 to smithi05931183-40 2024-04-01T13:30:33.781 INFO:tasks.rados.rados.0.smithi059.stdout:40: oid 40 updating version 0 to 11 2024-04-01T13:30:33.781 INFO:tasks.rados.rados.0.smithi059.stdout:40: oid 40 version 11 is already newer than 10 2024-04-01T13:30:33.781 INFO:tasks.rados.rados.0.smithi059.stdout:40: oid 40 updating version 11 to 12 2024-04-01T13:30:33.781 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 40 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-01T13:30:33.781 INFO:tasks.rados.rados.0.smithi059.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-04-01T13:30:33.786 INFO:tasks.rados.rados.0.smithi059.stdout:49: finishing write tid 1 to smithi05931183-49 2024-04-01T13:30:33.786 INFO:tasks.rados.rados.0.smithi059.stdout:42: finishing write tid 1 to smithi05931183-42 2024-04-01T13:30:33.820 INFO:tasks.rados.rados.0.smithi059.stdout:42: finishing write tid 2 to smithi05931183-42 2024-04-01T13:30:33.824 INFO:tasks.rados.rados.0.smithi059.stdout:49: finishing write tid 2 to smithi05931183-49 2024-04-01T13:30:33.824 INFO:tasks.rados.rados.0.smithi059.stdout:49: finishing write tid 3 to smithi05931183-49 2024-04-01T13:30:33.826 INFO:tasks.rados.rados.0.smithi059.stdout:49: finishing write tid 4 to smithi05931183-49 2024-04-01T13:30:33.826 INFO:tasks.rados.rados.0.smithi059.stdout:49: oid 49 updating version 0 to 19 2024-04-01T13:30:33.826 INFO:tasks.rados.rados.0.smithi059.stdout:49: oid 49 updating version 19 to 21 2024-04-01T13:30:33.826 INFO:tasks.rados.rados.0.smithi059.stdout:49: oid 49 version 21 is already newer than 21 2024-04-01T13:30:33.826 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 49 v 21 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-01T13:30:33.826 INFO:tasks.rados.rados.0.smithi059.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-04-01T13:30:33.827 INFO:tasks.rados.rados.0.smithi059.stdout:42: finishing write tid 3 to smithi05931183-42 2024-04-01T13:30:33.828 INFO:tasks.rados.rados.0.smithi059.stdout:42: finishing write tid 4 to smithi05931183-42 2024-04-01T13:30:33.828 INFO:tasks.rados.rados.0.smithi059.stdout:42: oid 42 updating version 0 to 19 2024-04-01T13:30:33.828 INFO:tasks.rados.rados.0.smithi059.stdout:42: oid 42 updating version 19 to 21 2024-04-01T13:30:33.829 INFO:tasks.rados.rados.0.smithi059.stdout:42: oid 42 version 21 is already newer than 21 2024-04-01T13:30:33.829 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 42 v 21 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-01T13:30:33.829 INFO:tasks.rados.rados.0.smithi059.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-04-01T13:30:33.845 INFO:tasks.rados.rados.0.smithi059.stdout:53: expect (ObjNum 19 snap 0 seq_num 19) 2024-04-01T13:30:33.941 INFO:tasks.rados.rados.0.smithi059.stdout:29: done (15 left) 2024-04-01T13:30:33.941 INFO:tasks.rados.rados.0.smithi059.stdout:34: done (14 left) 2024-04-01T13:30:33.941 INFO:tasks.rados.rados.0.smithi059.stdout:35: done (13 left) 2024-04-01T13:30:33.941 INFO:tasks.rados.rados.0.smithi059.stdout:39: done (12 left) 2024-04-01T13:30:33.941 INFO:tasks.rados.rados.0.smithi059.stdout:40: done (11 left) 2024-04-01T13:30:33.941 INFO:tasks.rados.rados.0.smithi059.stdout:42: done (10 left) 2024-04-01T13:30:33.941 INFO:tasks.rados.rados.0.smithi059.stdout:43: done (9 left) 2024-04-01T13:30:33.941 INFO:tasks.rados.rados.0.smithi059.stdout:44: done (8 left) 2024-04-01T13:30:33.942 INFO:tasks.rados.rados.0.smithi059.stdout:47: done (7 left) 2024-04-01T13:30:33.942 INFO:tasks.rados.rados.0.smithi059.stdout:48: done (6 left) 2024-04-01T13:30:33.942 INFO:tasks.rados.rados.0.smithi059.stdout:49: done (5 left) 2024-04-01T13:30:33.942 INFO:tasks.rados.rados.0.smithi059.stdout:50: done (4 left) 2024-04-01T13:30:33.942 INFO:tasks.rados.rados.0.smithi059.stdout:51: done (3 left) 2024-04-01T13:30:33.942 INFO:tasks.rados.rados.0.smithi059.stdout:52: done (2 left) 2024-04-01T13:30:33.942 INFO:tasks.rados.rados.0.smithi059.stdout:53: done (1 left) 2024-04-01T13:30:33.942 INFO:tasks.rados.rados.0.smithi059.stdout:54: done (0 left) 2024-04-01T13:30:33.942 INFO:tasks.rados.rados.0.smithi059.stdout:55: write oid 21 current snap is 0 2024-04-01T13:30:33.942 INFO:tasks.rados.rados.0.smithi059.stdout:55: seq_num 53 ranges {490771=686315,1885075=406346} 2024-04-01T13:30:33.947 INFO:tasks.rados.rados.0.smithi059.stdout:55: writing smithi05931183-21 from 490771 to 1177086 tid 1 2024-04-01T13:30:33.953 INFO:tasks.rados.rados.0.smithi059.stdout:55: writing smithi05931183-21 from 1885075 to 2291421 tid 2 2024-04-01T13:30:33.954 INFO:tasks.rados.rados.0.smithi059.stdout:56: write oid 26 current snap is 0 2024-04-01T13:30:33.954 INFO:tasks.rados.rados.0.smithi059.stdout:56: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2024-04-01T13:30:33.959 INFO:tasks.rados.rados.0.smithi059.stdout:56: writing smithi05931183-26 from 507578 to 985495 tid 1 2024-04-01T13:30:33.971 INFO:tasks.rados.rados.0.smithi059.stdout:56: writing smithi05931183-26 from 1543557 to 2093561 tid 2 2024-04-01T13:30:33.977 INFO:tasks.rados.rados.0.smithi059.stdout:56: writing smithi05931183-26 from 2833665 to 2907578 tid 3 2024-04-01T13:30:33.977 INFO:tasks.rados.rados.0.smithi059.stdout:57: write (excl) oid 13 current snap is 0 2024-04-01T13:30:33.977 INFO:tasks.rados.rados.0.smithi059.stdout:57: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-04-01T13:30:33.985 INFO:tasks.rados.rados.0.smithi059.stdout:57: writing smithi05931183-13 from 524385 to 1077551 tid 1 2024-04-01T13:30:33.991 INFO:tasks.rados.rados.0.smithi059.stdout:57: writing smithi05931183-13 from 1602039 to 2012054 tid 2 2024-04-01T13:30:33.997 INFO:tasks.rados.rados.0.smithi059.stdout:57: writing smithi05931183-13 from 2577441 to 2924385 tid 3 2024-04-01T13:30:33.997 INFO:tasks.rados.rados.0.smithi059.stdout:58: read oid 43 snap -1 2024-04-01T13:30:33.997 INFO:tasks.rados.rados.0.smithi059.stdout:58: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-01T13:30:33.997 INFO:tasks.rados.rados.0.smithi059.stdout:59: write (excl) oid 4 current snap is 0 2024-04-01T13:30:33.998 INFO:tasks.rados.rados.0.smithi059.stdout:55: finishing write tid 1 to smithi05931183-21 2024-04-01T13:30:33.998 INFO:tasks.rados.rados.0.smithi059.stdout:59: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2024-04-01T13:30:34.004 INFO:tasks.rados.rados.0.smithi059.stdout:59: writing smithi05931183-4 from 541192 to 1169607 tid 1 2024-04-01T13:30:34.009 INFO:tasks.rados.rados.0.smithi059.stdout:59: writing smithi05931183-4 from 1660521 to 2214194 tid 2 2024-04-01T13:30:34.014 INFO:tasks.rados.rados.0.smithi059.stdout:59: writing smithi05931183-4 from 2888511 to 2941192 tid 3 2024-04-01T13:30:34.014 INFO:tasks.rados.rados.0.smithi059.stdout:60: delete oid 48 current snap is 0 2024-04-01T13:30:34.014 INFO:tasks.rados.rados.0.smithi059.stdout:56: finishing write tid 1 to smithi05931183-26 2024-04-01T13:30:34.034 INFO:tasks.rados.rados.0.smithi059.stdout:55: finishing write tid 2 to smithi05931183-21 2024-04-01T13:30:34.039 INFO:tasks.rados.rados.0.smithi059.stdout:56: finishing write tid 2 to smithi05931183-26 2024-04-01T13:30:34.054 INFO:tasks.rados.rados.0.smithi059.stdout:57: finishing write tid 1 to smithi05931183-13 2024-04-01T13:30:34.069 INFO:tasks.rados.rados.0.smithi059.stdout:59: finishing write tid 1 to smithi05931183-4 2024-04-01T13:30:34.078 INFO:tasks.rados.rados.0.smithi059.stdout:55: finishing write tid 3 to smithi05931183-21 2024-04-01T13:30:34.084 INFO:tasks.rados.rados.0.smithi059.stdout:55: finishing write tid 4 to smithi05931183-21 2024-04-01T13:30:34.084 INFO:tasks.rados.rados.0.smithi059.stdout:55: oid 21 updating version 0 to 12 2024-04-01T13:30:34.084 INFO:tasks.rados.rados.0.smithi059.stdout:55: oid 21 version 12 is already newer than 11 2024-04-01T13:30:34.084 INFO:tasks.rados.rados.0.smithi059.stdout:55: oid 21 version 12 is already newer than 10 2024-04-01T13:30:34.084 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 21 v 12 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-04-01T13:30:34.084 INFO:tasks.rados.rados.0.smithi059.stdout:55: left oid 21 (ObjNum 53 snap 0 seq_num 53) 2024-04-01T13:30:34.087 INFO:tasks.rados.rados.0.smithi059.stdout:56: finishing write tid 3 to smithi05931183-26 2024-04-01T13:30:34.092 INFO:tasks.rados.rados.0.smithi059.stdout:57: finishing write tid 2 to smithi05931183-13 2024-04-01T13:30:34.127 INFO:tasks.rados.rados.0.smithi059.stdout:59: finishing write tid 2 to smithi05931183-4 2024-04-01T13:30:34.127 INFO:tasks.rados.rados.0.smithi059.stdout:56: finishing write tid 4 to smithi05931183-26 2024-04-01T13:30:34.127 INFO:tasks.rados.rados.0.smithi059.stdout:56: finishing write tid 5 to smithi05931183-26 2024-04-01T13:30:34.127 INFO:tasks.rados.rados.0.smithi059.stdout:56: oid 26 updating version 0 to 13 2024-04-01T13:30:34.127 INFO:tasks.rados.rados.0.smithi059.stdout:56: oid 26 updating version 13 to 16 2024-04-01T13:30:34.127 INFO:tasks.rados.rados.0.smithi059.stdout:56: oid 26 version 16 is already newer than 15 2024-04-01T13:30:34.127 INFO:tasks.rados.rados.0.smithi059.stdout:56: oid 26 version 16 is already newer than 14 2024-04-01T13:30:34.127 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 26 v 16 (ObjNum 54 snap 0 seq_num 54) dirty exists 2024-04-01T13:30:34.127 INFO:tasks.rados.rados.0.smithi059.stdout:56: left oid 26 (ObjNum 54 snap 0 seq_num 54) 2024-04-01T13:30:34.131 INFO:tasks.rados.rados.0.smithi059.stdout:57: finishing write tid 3 to smithi05931183-13 2024-04-01T13:30:34.132 INFO:tasks.rados.rados.0.smithi059.stdout:57: finishing write tid 4 to smithi05931183-13 2024-04-01T13:30:34.135 INFO:tasks.rados.rados.0.smithi059.stdout:58: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-01T13:30:34.242 INFO:tasks.rados.rados.0.smithi059.stdout:57: finishing write tid 5 to smithi05931183-13 2024-04-01T13:30:34.242 INFO:tasks.rados.rados.0.smithi059.stdout:57: oid 13 updating version 0 to 17 2024-04-01T13:30:34.242 INFO:tasks.rados.rados.0.smithi059.stdout:57: oid 13 updating version 17 to 19 2024-04-01T13:30:34.242 INFO:tasks.rados.rados.0.smithi059.stdout:57: oid 13 version 19 is already newer than 16 2024-04-01T13:30:34.242 INFO:tasks.rados.rados.0.smithi059.stdout:57: oid 13 version 19 is already newer than 19 2024-04-01T13:30:34.242 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 13 v 19 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-04-01T13:30:34.242 INFO:tasks.rados.rados.0.smithi059.stdout:57: left oid 13 (ObjNum 55 snap 0 seq_num 55) 2024-04-01T13:30:34.242 INFO:tasks.rados.rados.0.smithi059.stdout:59: finishing write tid 3 to smithi05931183-4 2024-04-01T13:30:34.242 INFO:tasks.rados.rados.0.smithi059.stdout:55: done (5 left) 2024-04-01T13:30:34.242 INFO:tasks.rados.rados.0.smithi059.stdout:56: done (4 left) 2024-04-01T13:30:34.243 INFO:tasks.rados.rados.0.smithi059.stdout:57: done (3 left) 2024-04-01T13:30:34.243 INFO:tasks.rados.rados.0.smithi059.stdout:58: done (2 left) 2024-04-01T13:30:34.243 INFO:tasks.rados.rados.0.smithi059.stdout:60: done (1 left) 2024-04-01T13:30:34.243 INFO:tasks.rados.rados.0.smithi059.stdout:61: delete oid 1 current snap is 0 2024-04-01T13:30:34.243 INFO:tasks.rados.rados.0.smithi059.stdout:59: finishing write tid 4 to smithi05931183-4 2024-04-01T13:30:34.243 INFO:tasks.rados.rados.0.smithi059.stdout:59: finishing write tid 5 to smithi05931183-4 2024-04-01T13:30:34.243 INFO:tasks.rados.rados.0.smithi059.stdout:59: oid 4 updating version 0 to 13 2024-04-01T13:30:34.243 INFO:tasks.rados.rados.0.smithi059.stdout:59: oid 4 updating version 13 to 14 2024-04-01T13:30:34.243 INFO:tasks.rados.rados.0.smithi059.stdout:59: oid 4 updating version 14 to 16 2024-04-01T13:30:34.243 INFO:tasks.rados.rados.0.smithi059.stdout:59: oid 4 version 16 is already newer than 15 2024-04-01T13:30:34.243 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 4 v 16 (ObjNum 56 snap 0 seq_num 56) dirty exists 2024-04-01T13:30:34.243 INFO:tasks.rados.rados.0.smithi059.stdout:59: left oid 4 (ObjNum 56 snap 0 seq_num 56) 2024-04-01T13:30:34.257 INFO:tasks.rados.rados.0.smithi059.stdout:59: done (1 left) 2024-04-01T13:30:34.257 INFO:tasks.rados.rados.0.smithi059.stdout:61: done (0 left) 2024-04-01T13:30:34.257 INFO:tasks.rados.rados.0.smithi059.stdout:62: delete oid 27 current snap is 0 2024-04-01T13:30:34.275 INFO:tasks.rados.rados.0.smithi059.stdout:62: done (0 left) 2024-04-01T13:30:34.275 INFO:tasks.rados.rados.0.smithi059.stdout:63: read oid 15 snap -1 2024-04-01T13:30:34.275 INFO:tasks.rados.rados.0.smithi059.stdout:63: expect (ObjNum 14 snap 0 seq_num 14) 2024-04-01T13:30:34.275 INFO:tasks.rados.rados.0.smithi059.stdout:64: snap_create 2024-04-01T13:30:34.275 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 1 2024-04-01T13:30:34.295 INFO:tasks.rados.rados.0.smithi059.stdout:63: expect (ObjNum 14 snap 0 seq_num 14) 2024-04-01T13:30:34.316 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-01T13:30:34.316 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-01T13:30:34.316 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 2, 3] 2024-04-01T13:30:34.317 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-01T13:30:34.317 INFO:teuthology.orchestra.run:waiting for 300 2024-04-01T13:30:34.374 INFO:tasks.ceph.osd.3:Stopped 2024-04-01T13:30:34.384 INFO:tasks.rados.rados.0.smithi059.stdout:63: done (0 left) 2024-04-01T13:30:34.530 INFO:tasks.rados.rados.0.smithi059.stdout:64: done (0 left) 2024-04-01T13:30:34.530 INFO:tasks.rados.rados.0.smithi059.stdout:65: write oid 40 current snap is 1 2024-04-01T13:30:34.530 INFO:tasks.rados.rados.0.smithi059.stdout:65: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-04-01T13:30:34.536 INFO:tasks.rados.rados.0.smithi059.stdout:65: writing smithi05931183-40 from 608420 to 1137831 tid 1 2024-04-01T13:30:34.542 INFO:tasks.rados.rados.0.smithi059.stdout:65: writing smithi05931183-40 from 1778096 to 2339107 tid 2 2024-04-01T13:30:34.547 INFO:tasks.rados.rados.0.smithi059.stdout:65: writing smithi05931183-40 from 2881850 to 3008420 tid 3 2024-04-01T13:30:34.547 INFO:tasks.rados.rados.0.smithi059.stdout:66: read oid 48 snap 0 2024-04-01T13:30:34.547 INFO:tasks.rados.rados.0.smithi059.stdout:66: expect deleted 2024-04-01T13:30:34.547 INFO:tasks.rados.rados.0.smithi059.stdout:67: read oid 28 snap 0 2024-04-01T13:30:34.547 INFO:tasks.rados.rados.0.smithi059.stdout:67: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-01T13:30:34.548 INFO:tasks.rados.rados.0.smithi059.stdout:68: read oid 13 snap -1 2024-04-01T13:30:34.548 INFO:tasks.rados.rados.0.smithi059.stdout:68: expect (ObjNum 55 snap 0 seq_num 55) 2024-04-01T13:30:34.548 INFO:tasks.rados.rados.0.smithi059.stdout:69: read oid 43 snap -1 2024-04-01T13:30:34.548 INFO:tasks.rados.rados.0.smithi059.stdout:69: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-01T13:30:34.548 INFO:tasks.rados.rados.0.smithi059.stdout:70: read oid 36 snap -1 2024-04-01T13:30:34.548 INFO:tasks.rados.rados.0.smithi059.stdout:70: expect (ObjNum 35 snap 0 seq_num 35) 2024-04-01T13:30:34.548 INFO:tasks.rados.rados.0.smithi059.stdout:71: read oid 50 snap -1 2024-04-01T13:30:34.548 INFO:tasks.rados.rados.0.smithi059.stdout:71: expect (ObjNum 49 snap 0 seq_num 49) 2024-04-01T13:30:34.548 INFO:tasks.rados.rados.0.smithi059.stdout:72: read oid 8 snap 0 2024-04-01T13:30:34.548 INFO:tasks.rados.rados.0.smithi059.stdout:72: expect (ObjNum 7 snap 0 seq_num 7) 2024-04-01T13:30:34.548 INFO:tasks.rados.rados.0.smithi059.stdout:73: read oid 27 snap 0 2024-04-01T13:30:34.548 INFO:tasks.rados.rados.0.smithi059.stdout:73: expect deleted 2024-04-01T13:30:34.548 INFO:tasks.rados.rados.0.smithi059.stdout:74: snap_create 2024-04-01T13:30:34.548 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 9 2024-04-01T13:30:34.584 INFO:tasks.rados.rados.0.smithi059.stdout:73: done (8 left) 2024-04-01T13:30:34.584 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 8 2024-04-01T13:30:34.619 INFO:tasks.rados.rados.0.smithi059.stdout:70: expect (ObjNum 35 snap 0 seq_num 35) 2024-04-01T13:30:34.720 INFO:tasks.rados.rados.0.smithi059.stdout:70: done (7 left) 2024-04-01T13:30:34.720 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 7 2024-04-01T13:30:35.833 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T13:30:35.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:35.969+0000 7ff82d79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:35.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:35.976+0000 7ff82d79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:39.374 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2] 2024-04-01T13:30:39.375 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-01T13:30:39.375 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-04-01T13:30:39.375 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-01T13:30:39.375 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-01T13:30:39.378 INFO:tasks.ceph.osd.3:Started 2024-04-01T13:30:39.378 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:30:39.489 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:30:39.492 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:30:39.493 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:30:43.547 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-01T13:30:43.551 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-01T13:30:43.551 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T13:30:44.494 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:30:44.657 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-01T13:30:44.657 INFO:teuthology.orchestra.run.smithi087.stdout: "ops_in_flight": [ 2024-04-01T13:30:44.658 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.658 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "peering_event", 2024-04-01T13:30:44.658 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 43, 2024-04-01T13:30:44.658 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.658 INFO:teuthology.orchestra.run.smithi087.stdout: "PeeringEvent": { 2024-04-01T13:30:44.658 INFO:teuthology.orchestra.run.smithi087.stdout: "from": "1", 2024-04-01T13:30:44.658 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "3.0", 2024-04-01T13:30:44.658 INFO:teuthology.orchestra.run.smithi087.stdout: "sent": 20, 2024-04-01T13:30:44.658 INFO:teuthology.orchestra.run.smithi087.stdout: "requested": 20, 2024-04-01T13:30:44.658 INFO:teuthology.orchestra.run.smithi087.stdout: "evt": "epoch_sent: 20 epoch_requested: 20 MNotifyRec 3.0 from 1 notify: (query:20 sent:20 3.0( v 16'22 (0'0,16'22] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=20 pruub=2.161452055s) ([14,19] all_participants={1, 2, 3} intervals=[([14,18] acting={1, 2, 3})])) features: 0x3f03cffffffdffff +create_info", 2024-04-01T13:30:44.658 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-04-01T13:30:44.658 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.658 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:30:44.659 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.648402+0000" 2024-04-01T13:30:44.659 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.659 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.659 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:30:44.659 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.648420+0000" 2024-04-01T13:30:44.659 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.659 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.659 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:30:44.659 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.650695+0000" 2024-04-01T13:30:44.659 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.659 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.659 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:30:44.659 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.653719+0000", 2024-04-01T13:30:44.659 INFO:teuthology.orchestra.run.smithi087.stdout: "blocker": { 2024-04-01T13:30:44.659 INFO:teuthology.orchestra.run.smithi087.stdout: "op_type": "PerShardState::osdmap_gate", 2024-04-01T13:30:44.660 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.660 INFO:teuthology.orchestra.run.smithi087.stdout: "OSDMapGate": { 2024-04-01T13:30:44.660 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 20 2024-04-01T13:30:44.660 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.660 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.660 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.660 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.660 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.660 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:30:44.660 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.660 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.660 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.660 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:30:44.660 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.660 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.661 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.661 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:30:44.661 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.661 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.661 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.661 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-01T13:30:44.661 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.661 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.661 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.661 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-01T13:30:44.661 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.661 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.661 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.661 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-01T13:30:44.662 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.662 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.662 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.662 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:30:44.662 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.662 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.662 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-01T13:30:44.662 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.662 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.662 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.662 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.662 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "peering_event", 2024-04-01T13:30:44.662 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 44, 2024-04-01T13:30:44.662 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.663 INFO:teuthology.orchestra.run.smithi087.stdout: "PeeringEvent": { 2024-04-01T13:30:44.663 INFO:teuthology.orchestra.run.smithi087.stdout: "from": "2", 2024-04-01T13:30:44.663 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "2.5", 2024-04-01T13:30:44.663 INFO:teuthology.orchestra.run.smithi087.stdout: "sent": 20, 2024-04-01T13:30:44.663 INFO:teuthology.orchestra.run.smithi087.stdout: "requested": 20, 2024-04-01T13:30:44.663 INFO:teuthology.orchestra.run.smithi087.stdout: "evt": "epoch_sent: 20 epoch_requested: 20 MNotifyRec 2.5 from 2 notify: (query:20 sent:20 2.5( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=20 pruub=3.893759966s) ([11,19] all_participants={0, 2, 3} intervals=[([11,18] acting={0, 2, 3})])) features: 0x3f03cffffffdffff +create_info", 2024-04-01T13:30:44.663 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-04-01T13:30:44.663 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.663 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:30:44.663 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.657580+0000" 2024-04-01T13:30:44.663 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.663 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.663 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:30:44.663 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.657597+0000" 2024-04-01T13:30:44.664 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.664 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.664 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:30:44.664 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.657759+0000" 2024-04-01T13:30:44.664 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.664 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.664 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:30:44.664 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.658061+0000", 2024-04-01T13:30:44.664 INFO:teuthology.orchestra.run.smithi087.stdout: "blocker": { 2024-04-01T13:30:44.664 INFO:teuthology.orchestra.run.smithi087.stdout: "op_type": "PerShardState::osdmap_gate", 2024-04-01T13:30:44.664 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.664 INFO:teuthology.orchestra.run.smithi087.stdout: "OSDMapGate": { 2024-04-01T13:30:44.664 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 20 2024-04-01T13:30:44.664 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.664 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.665 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.665 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.665 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.665 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:30:44.665 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.665 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.665 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.665 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:30:44.665 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.665 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.665 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.665 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:30:44.665 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.665 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.665 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.665 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-01T13:30:44.666 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.666 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.666 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.666 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-01T13:30:44.666 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.666 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.666 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.666 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-01T13:30:44.666 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.666 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.666 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.666 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:30:44.666 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.666 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.667 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-01T13:30:44.667 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.667 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.667 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.667 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.667 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "peering_event", 2024-04-01T13:30:44.667 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 45, 2024-04-01T13:30:44.667 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.667 INFO:teuthology.orchestra.run.smithi087.stdout: "PeeringEvent": { 2024-04-01T13:30:44.667 INFO:teuthology.orchestra.run.smithi087.stdout: "from": "1", 2024-04-01T13:30:44.667 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "3.1", 2024-04-01T13:30:44.667 INFO:teuthology.orchestra.run.smithi087.stdout: "sent": 20, 2024-04-01T13:30:44.667 INFO:teuthology.orchestra.run.smithi087.stdout: "requested": 20, 2024-04-01T13:30:44.667 INFO:teuthology.orchestra.run.smithi087.stdout: "evt": "epoch_sent: 20 epoch_requested: 20 MNotifyRec 3.1 from 1 notify: (query:20 sent:20 3.1( v 16'12 (0'0,16'12] local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=20 pruub=2.091959238s) ([14,19] all_participants={0, 1, 3} intervals=[([14,18] acting={0, 1, 3})])) features: 0x3f03cffffffdffff +create_info", 2024-04-01T13:30:44.668 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-04-01T13:30:44.668 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.668 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:30:44.668 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.661325+0000" 2024-04-01T13:30:44.668 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.668 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.668 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:30:44.668 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.661333+0000" 2024-04-01T13:30:44.668 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.668 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.668 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:30:44.668 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.663209+0000", 2024-04-01T13:30:44.668 INFO:teuthology.orchestra.run.smithi087.stdout: "blocker": { 2024-04-01T13:30:44.668 INFO:teuthology.orchestra.run.smithi087.stdout: "op_type": "ConnectionPipeline::await_map", 2024-04-01T13:30:44.669 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.669 INFO:teuthology.orchestra.run.smithi087.stdout: "waiting": 1, 2024-04-01T13:30:44.669 INFO:teuthology.orchestra.run.smithi087.stdout: "held_by_operation_id": 43 2024-04-01T13:30:44.669 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.669 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.669 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.669 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.669 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:30:44.669 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.669 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.669 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.669 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:30:44.669 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.669 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.669 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.670 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:30:44.670 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.670 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.670 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.670 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:30:44.670 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.670 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.670 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.670 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-01T13:30:44.670 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.670 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.670 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.670 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-01T13:30:44.670 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.671 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.671 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.671 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-01T13:30:44.671 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.671 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.671 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.671 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:30:44.671 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.671 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.671 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-01T13:30:44.671 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.671 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.671 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.671 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.672 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "peering_event", 2024-04-01T13:30:44.672 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 46, 2024-04-01T13:30:44.672 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.672 INFO:teuthology.orchestra.run.smithi087.stdout: "PeeringEvent": { 2024-04-01T13:30:44.672 INFO:teuthology.orchestra.run.smithi087.stdout: "from": "2", 2024-04-01T13:30:44.672 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "1.0", 2024-04-01T13:30:44.672 INFO:teuthology.orchestra.run.smithi087.stdout: "sent": 20, 2024-04-01T13:30:44.672 INFO:teuthology.orchestra.run.smithi087.stdout: "requested": 20, 2024-04-01T13:30:44.672 INFO:teuthology.orchestra.run.smithi087.stdout: "evt": "epoch_sent: 20 epoch_requested: 20 MNotifyRec 1.0 from 2 notify: (query:20 sent:20 1.0( v 12'97 (0'0,12'97] local-lis/les=10/11 n=0 ec=10/10 lis/c=10/10 les/c/f=11/11/0 sis=20 pruub=2.864842415s) ([10,19] all_participants={0, 2, 3} intervals=[([10,18] acting={0, 2, 3})])) features: 0x3f03cffffffdffff +create_info", 2024-04-01T13:30:44.672 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-04-01T13:30:44.672 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.672 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:30:44.672 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.663762+0000" 2024-04-01T13:30:44.672 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.673 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.673 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:30:44.673 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.663769+0000" 2024-04-01T13:30:44.673 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.673 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.673 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:30:44.673 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.664102+0000", 2024-04-01T13:30:44.673 INFO:teuthology.orchestra.run.smithi087.stdout: "blocker": { 2024-04-01T13:30:44.673 INFO:teuthology.orchestra.run.smithi087.stdout: "op_type": "ConnectionPipeline::await_map", 2024-04-01T13:30:44.673 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.673 INFO:teuthology.orchestra.run.smithi087.stdout: "waiting": 1, 2024-04-01T13:30:44.673 INFO:teuthology.orchestra.run.smithi087.stdout: "held_by_operation_id": 44 2024-04-01T13:30:44.673 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.673 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.673 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.674 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.674 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:30:44.674 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.674 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.674 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.674 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:30:44.674 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.674 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.674 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.674 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:30:44.674 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.674 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.674 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.674 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:30:44.674 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.675 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.675 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.675 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-01T13:30:44.675 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.675 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.675 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.675 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-01T13:30:44.675 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.675 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.675 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.675 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-01T13:30:44.675 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.675 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.675 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.675 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:30:44.676 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.676 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.676 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-01T13:30:44.676 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.676 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.676 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.676 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.676 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "peering_event", 2024-04-01T13:30:44.676 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 16777247, 2024-04-01T13:30:44.676 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.676 INFO:teuthology.orchestra.run.smithi087.stdout: "PeeringEvent": { 2024-04-01T13:30:44.676 INFO:teuthology.orchestra.run.smithi087.stdout: "from": "3", 2024-04-01T13:30:44.676 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "2.1", 2024-04-01T13:30:44.676 INFO:teuthology.orchestra.run.smithi087.stdout: "sent": 20, 2024-04-01T13:30:44.676 INFO:teuthology.orchestra.run.smithi087.stdout: "requested": 20, 2024-04-01T13:30:44.677 INFO:teuthology.orchestra.run.smithi087.stdout: "evt": "epoch_sent: 20 epoch_requested: 20 IntervalFlush", 2024-04-01T13:30:44.677 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-04-01T13:30:44.677 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.677 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:30:44.677 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.677 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.677 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.677 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-01T13:30:44.677 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.647252+0000" 2024-04-01T13:30:44.677 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.677 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.677 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-01T13:30:44.677 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.677 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.678 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.678 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-01T13:30:44.678 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.648592+0000" 2024-04-01T13:30:44.678 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.678 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.678 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:30:44.678 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.678 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.678 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-01T13:30:44.678 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.678 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.678 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.678 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.678 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "peering_event", 2024-04-01T13:30:44.679 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 16777248, 2024-04-01T13:30:44.679 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.679 INFO:teuthology.orchestra.run.smithi087.stdout: "PeeringEvent": { 2024-04-01T13:30:44.679 INFO:teuthology.orchestra.run.smithi087.stdout: "from": "3", 2024-04-01T13:30:44.679 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "3.6", 2024-04-01T13:30:44.679 INFO:teuthology.orchestra.run.smithi087.stdout: "sent": 20, 2024-04-01T13:30:44.679 INFO:teuthology.orchestra.run.smithi087.stdout: "requested": 20, 2024-04-01T13:30:44.679 INFO:teuthology.orchestra.run.smithi087.stdout: "evt": "epoch_sent: 20 epoch_requested: 20 IntervalFlush", 2024-04-01T13:30:44.679 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-04-01T13:30:44.679 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.679 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:30:44.679 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.679 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.679 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.679 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-01T13:30:44.680 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.647107+0000" 2024-04-01T13:30:44.680 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.680 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.680 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-01T13:30:44.680 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.680 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.680 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.680 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-01T13:30:44.680 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.647648+0000" 2024-04-01T13:30:44.680 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.680 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.680 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:30:44.680 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.680 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.680 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-01T13:30:44.681 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.681 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.681 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.681 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.681 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "peering_event", 2024-04-01T13:30:44.681 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 16777249, 2024-04-01T13:30:44.681 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.681 INFO:teuthology.orchestra.run.smithi087.stdout: "PeeringEvent": { 2024-04-01T13:30:44.681 INFO:teuthology.orchestra.run.smithi087.stdout: "from": "3", 2024-04-01T13:30:44.681 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "3.8", 2024-04-01T13:30:44.681 INFO:teuthology.orchestra.run.smithi087.stdout: "sent": 20, 2024-04-01T13:30:44.681 INFO:teuthology.orchestra.run.smithi087.stdout: "requested": 20, 2024-04-01T13:30:44.681 INFO:teuthology.orchestra.run.smithi087.stdout: "evt": "epoch_sent: 20 epoch_requested: 20 IntervalFlush", 2024-04-01T13:30:44.682 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-04-01T13:30:44.682 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.682 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:30:44.682 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.682 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.682 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.682 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-01T13:30:44.682 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.647830+0000" 2024-04-01T13:30:44.682 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.682 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.682 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-01T13:30:44.682 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.682 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.682 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.682 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-01T13:30:44.683 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.648454+0000" 2024-04-01T13:30:44.683 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.683 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.683 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:30:44.683 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.683 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.683 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-01T13:30:44.683 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.683 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.683 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.683 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.683 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "peering_event", 2024-04-01T13:30:44.683 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 16777250, 2024-04-01T13:30:44.683 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.684 INFO:teuthology.orchestra.run.smithi087.stdout: "PeeringEvent": { 2024-04-01T13:30:44.684 INFO:teuthology.orchestra.run.smithi087.stdout: "from": "3", 2024-04-01T13:30:44.684 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "2.5", 2024-04-01T13:30:44.684 INFO:teuthology.orchestra.run.smithi087.stdout: "sent": 20, 2024-04-01T13:30:44.684 INFO:teuthology.orchestra.run.smithi087.stdout: "requested": 20, 2024-04-01T13:30:44.684 INFO:teuthology.orchestra.run.smithi087.stdout: "evt": "epoch_sent: 20 epoch_requested: 20 IntervalFlush", 2024-04-01T13:30:44.684 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-04-01T13:30:44.684 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.684 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:30:44.684 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.684 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.684 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.684 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-01T13:30:44.684 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.648083+0000" 2024-04-01T13:30:44.684 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.685 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.685 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-01T13:30:44.685 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.685 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.685 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.685 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-01T13:30:44.685 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.648502+0000" 2024-04-01T13:30:44.685 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.685 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.685 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:30:44.685 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.685 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.685 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-01T13:30:44.685 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.685 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.686 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.686 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.686 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "peering_event", 2024-04-01T13:30:44.686 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 16777251, 2024-04-01T13:30:44.686 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.686 INFO:teuthology.orchestra.run.smithi087.stdout: "PeeringEvent": { 2024-04-01T13:30:44.686 INFO:teuthology.orchestra.run.smithi087.stdout: "from": "3", 2024-04-01T13:30:44.686 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "3.3", 2024-04-01T13:30:44.686 INFO:teuthology.orchestra.run.smithi087.stdout: "sent": 20, 2024-04-01T13:30:44.686 INFO:teuthology.orchestra.run.smithi087.stdout: "requested": 20, 2024-04-01T13:30:44.686 INFO:teuthology.orchestra.run.smithi087.stdout: "evt": "epoch_sent: 20 epoch_requested: 20 IntervalFlush", 2024-04-01T13:30:44.686 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-04-01T13:30:44.686 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.686 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:30:44.687 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.687 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.687 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.687 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-01T13:30:44.687 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.648232+0000" 2024-04-01T13:30:44.687 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.687 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.687 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-01T13:30:44.687 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.687 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.687 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.687 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-01T13:30:44.687 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.648638+0000" 2024-04-01T13:30:44.687 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.687 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.688 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:30:44.688 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.688 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.688 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-01T13:30:44.688 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.688 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.688 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.688 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.688 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "peering_event", 2024-04-01T13:30:44.688 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 16777252, 2024-04-01T13:30:44.688 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.688 INFO:teuthology.orchestra.run.smithi087.stdout: "PeeringEvent": { 2024-04-01T13:30:44.688 INFO:teuthology.orchestra.run.smithi087.stdout: "from": "3", 2024-04-01T13:30:44.688 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "3.e", 2024-04-01T13:30:44.689 INFO:teuthology.orchestra.run.smithi087.stdout: "sent": 20, 2024-04-01T13:30:44.689 INFO:teuthology.orchestra.run.smithi087.stdout: "requested": 20, 2024-04-01T13:30:44.689 INFO:teuthology.orchestra.run.smithi087.stdout: "evt": "epoch_sent: 20 epoch_requested: 20 IntervalFlush", 2024-04-01T13:30:44.689 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-04-01T13:30:44.689 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.689 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:30:44.689 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.689 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.689 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.689 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-01T13:30:44.689 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.648416+0000" 2024-04-01T13:30:44.689 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.689 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.689 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-01T13:30:44.689 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.690 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.690 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.690 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-01T13:30:44.690 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.648575+0000" 2024-04-01T13:30:44.690 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.690 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.690 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:30:44.690 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.690 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.690 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-01T13:30:44.690 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.690 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.690 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.690 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.691 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "peering_event", 2024-04-01T13:30:44.691 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 33554463, 2024-04-01T13:30:44.691 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.691 INFO:teuthology.orchestra.run.smithi087.stdout: "PeeringEvent": { 2024-04-01T13:30:44.691 INFO:teuthology.orchestra.run.smithi087.stdout: "from": "3", 2024-04-01T13:30:44.691 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "3.5", 2024-04-01T13:30:44.691 INFO:teuthology.orchestra.run.smithi087.stdout: "sent": 20, 2024-04-01T13:30:44.691 INFO:teuthology.orchestra.run.smithi087.stdout: "requested": 20, 2024-04-01T13:30:44.691 INFO:teuthology.orchestra.run.smithi087.stdout: "evt": "epoch_sent: 20 epoch_requested: 20 IntervalFlush", 2024-04-01T13:30:44.691 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-04-01T13:30:44.691 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.691 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:30:44.691 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.691 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.691 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.692 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-01T13:30:44.692 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.650476+0000" 2024-04-01T13:30:44.692 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.692 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.692 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-01T13:30:44.692 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.692 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.692 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.692 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-01T13:30:44.692 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.650737+0000" 2024-04-01T13:30:44.692 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.692 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.692 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:30:44.692 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.693 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.693 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-01T13:30:44.693 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.693 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.693 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.693 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.693 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "peering_event", 2024-04-01T13:30:44.693 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 33554464, 2024-04-01T13:30:44.693 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.693 INFO:teuthology.orchestra.run.smithi087.stdout: "PeeringEvent": { 2024-04-01T13:30:44.693 INFO:teuthology.orchestra.run.smithi087.stdout: "from": "3", 2024-04-01T13:30:44.693 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "3.d", 2024-04-01T13:30:44.693 INFO:teuthology.orchestra.run.smithi087.stdout: "sent": 20, 2024-04-01T13:30:44.693 INFO:teuthology.orchestra.run.smithi087.stdout: "requested": 20, 2024-04-01T13:30:44.693 INFO:teuthology.orchestra.run.smithi087.stdout: "evt": "epoch_sent: 20 epoch_requested: 20 IntervalFlush", 2024-04-01T13:30:44.694 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-04-01T13:30:44.694 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.694 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:30:44.694 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.694 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.694 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.694 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-01T13:30:44.694 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.650351+0000" 2024-04-01T13:30:44.694 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.694 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.694 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-01T13:30:44.694 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.694 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.695 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.695 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-01T13:30:44.695 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.650886+0000" 2024-04-01T13:30:44.695 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.695 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.695 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:30:44.695 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.695 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.695 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-01T13:30:44.695 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.695 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.695 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.695 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.695 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "peering_event", 2024-04-01T13:30:44.695 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 33554465, 2024-04-01T13:30:44.696 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.696 INFO:teuthology.orchestra.run.smithi087.stdout: "PeeringEvent": { 2024-04-01T13:30:44.696 INFO:teuthology.orchestra.run.smithi087.stdout: "from": "3", 2024-04-01T13:30:44.696 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "3.c", 2024-04-01T13:30:44.696 INFO:teuthology.orchestra.run.smithi087.stdout: "sent": 20, 2024-04-01T13:30:44.696 INFO:teuthology.orchestra.run.smithi087.stdout: "requested": 20, 2024-04-01T13:30:44.696 INFO:teuthology.orchestra.run.smithi087.stdout: "evt": "epoch_sent: 20 epoch_requested: 20 IntervalFlush", 2024-04-01T13:30:44.696 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-04-01T13:30:44.696 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.696 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:30:44.696 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.696 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.696 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.696 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-01T13:30:44.696 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.650698+0000" 2024-04-01T13:30:44.697 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.697 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.697 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-01T13:30:44.697 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.697 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.697 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.697 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-01T13:30:44.697 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.651120+0000" 2024-04-01T13:30:44.697 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.697 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.697 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:30:44.697 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.697 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.697 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-01T13:30:44.697 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.698 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.698 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.698 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.698 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "peering_event", 2024-04-01T13:30:44.698 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 33554466, 2024-04-01T13:30:44.698 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.698 INFO:teuthology.orchestra.run.smithi087.stdout: "PeeringEvent": { 2024-04-01T13:30:44.698 INFO:teuthology.orchestra.run.smithi087.stdout: "from": "3", 2024-04-01T13:30:44.698 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "3.b", 2024-04-01T13:30:44.698 INFO:teuthology.orchestra.run.smithi087.stdout: "sent": 20, 2024-04-01T13:30:44.698 INFO:teuthology.orchestra.run.smithi087.stdout: "requested": 20, 2024-04-01T13:30:44.698 INFO:teuthology.orchestra.run.smithi087.stdout: "evt": "epoch_sent: 20 epoch_requested: 20 IntervalFlush", 2024-04-01T13:30:44.698 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-04-01T13:30:44.699 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.699 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:30:44.699 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.699 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.699 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.699 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-01T13:30:44.699 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.651072+0000" 2024-04-01T13:30:44.699 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.699 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.699 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-01T13:30:44.699 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.699 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.699 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.699 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-01T13:30:44.699 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.651161+0000" 2024-04-01T13:30:44.700 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.700 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.700 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:30:44.700 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.700 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.701 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-01T13:30:44.701 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.701 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.701 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.701 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.701 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "peering_event", 2024-04-01T13:30:44.702 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 33554467, 2024-04-01T13:30:44.702 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.702 INFO:teuthology.orchestra.run.smithi087.stdout: "PeeringEvent": { 2024-04-01T13:30:44.702 INFO:teuthology.orchestra.run.smithi087.stdout: "from": "3", 2024-04-01T13:30:44.702 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "2.4", 2024-04-01T13:30:44.702 INFO:teuthology.orchestra.run.smithi087.stdout: "sent": 20, 2024-04-01T13:30:44.702 INFO:teuthology.orchestra.run.smithi087.stdout: "requested": 20, 2024-04-01T13:30:44.702 INFO:teuthology.orchestra.run.smithi087.stdout: "evt": "epoch_sent: 20 epoch_requested: 20 IntervalFlush", 2024-04-01T13:30:44.702 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-04-01T13:30:44.702 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.702 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:30:44.702 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.702 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.702 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.703 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-01T13:30:44.703 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.654808+0000" 2024-04-01T13:30:44.703 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.703 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.703 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-01T13:30:44.703 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.703 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.703 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.703 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-01T13:30:44.703 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.654889+0000" 2024-04-01T13:30:44.703 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.703 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.703 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:30:44.703 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.703 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.704 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-01T13:30:44.704 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.704 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.704 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.704 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.704 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "peering_event", 2024-04-01T13:30:44.704 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 33554468, 2024-04-01T13:30:44.704 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:30:44.704 INFO:teuthology.orchestra.run.smithi087.stdout: "PeeringEvent": { 2024-04-01T13:30:44.704 INFO:teuthology.orchestra.run.smithi087.stdout: "from": "3", 2024-04-01T13:30:44.704 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "3.1", 2024-04-01T13:30:44.704 INFO:teuthology.orchestra.run.smithi087.stdout: "sent": 20, 2024-04-01T13:30:44.704 INFO:teuthology.orchestra.run.smithi087.stdout: "requested": 20, 2024-04-01T13:30:44.704 INFO:teuthology.orchestra.run.smithi087.stdout: "evt": "epoch_sent: 20 epoch_requested: 20 IntervalFlush", 2024-04-01T13:30:44.705 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-04-01T13:30:44.705 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.705 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:30:44.705 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.705 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.705 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.705 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-01T13:30:44.705 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.657945+0000" 2024-04-01T13:30:44.705 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.705 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.705 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-01T13:30:44.705 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.705 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.705 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.705 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-01T13:30:44.706 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:30:44.658282+0000" 2024-04-01T13:30:44.706 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:30:44.706 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:30:44.706 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:30:44.706 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:30:44.706 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.706 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-01T13:30:44.706 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.706 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.706 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:30:44.706 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-01T13:30:44.706 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-01T13:30:45.552 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T13:30:45.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:45.725+0000 7f674586d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:45.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:45.734+0000 7f674586d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:45.843 INFO:tasks.rados.rados.0.smithi059.stdout:65: finishing write tid 1 to smithi05931183-40 2024-04-01T13:30:45.881 INFO:tasks.rados.rados.0.smithi059.stdout:65: finishing write tid 2 to smithi05931183-40 2024-04-01T13:30:45.942 INFO:tasks.rados.rados.0.smithi059.stdout:65: finishing write tid 3 to smithi05931183-40 2024-04-01T13:30:45.953 INFO:tasks.rados.rados.0.smithi059.stdout:65: finishing write tid 4 to smithi05931183-40 2024-04-01T13:30:45.954 INFO:tasks.rados.rados.0.smithi059.stdout:65: finishing write tid 5 to smithi05931183-40 2024-04-01T13:30:45.954 INFO:tasks.rados.rados.0.smithi059.stdout:65: oid 40 updating version 0 to 20 2024-04-01T13:30:45.954 INFO:tasks.rados.rados.0.smithi059.stdout:65: oid 40 updating version 20 to 21 2024-04-01T13:30:45.954 INFO:tasks.rados.rados.0.smithi059.stdout:65: oid 40 version 21 is already newer than 19 2024-04-01T13:30:45.954 INFO:tasks.rados.rados.0.smithi059.stdout:65: oid 40 version 21 is already newer than 0 2024-04-01T13:30:45.954 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 40 v 21 (ObjNum 60 snap 1 seq_num 60) dirty exists 2024-04-01T13:30:45.955 INFO:tasks.rados.rados.0.smithi059.stdout:65: left oid 40 (ObjNum 60 snap 1 seq_num 60) 2024-04-01T13:30:45.955 INFO:tasks.rados.rados.0.smithi059.stdout:65: done (6 left) 2024-04-01T13:30:45.955 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 6 2024-04-01T13:30:46.599 INFO:tasks.rados.rados.0.smithi059.stdout:66: done (5 left) 2024-04-01T13:30:46.599 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 5 2024-04-01T13:30:46.652 INFO:tasks.rados.rados.0.smithi059.stdout:67: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-01T13:30:46.753 INFO:tasks.rados.rados.0.smithi059.stdout:67: done (4 left) 2024-04-01T13:30:46.754 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 4 2024-04-01T13:30:46.754 INFO:tasks.rados.rados.0.smithi059.stdout:71: expect (ObjNum 49 snap 0 seq_num 49) 2024-04-01T13:30:46.861 INFO:tasks.rados.rados.0.smithi059.stdout:71: done (3 left) 2024-04-01T13:30:46.861 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 3 2024-04-01T13:30:46.861 INFO:tasks.rados.rados.0.smithi059.stdout:72: expect (ObjNum 7 snap 0 seq_num 7) 2024-04-01T13:30:46.943 INFO:tasks.rados.rados.0.smithi059.stdout:69: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-01T13:30:47.048 INFO:tasks.rados.rados.0.smithi059.stdout:68: expect (ObjNum 55 snap 0 seq_num 55) 2024-04-01T13:30:47.194 INFO:tasks.rados.rados.0.smithi059.stdout:68: done (2 left) 2024-04-01T13:30:47.194 INFO:tasks.rados.rados.0.smithi059.stdout:69: done (1 left) 2024-04-01T13:30:47.195 INFO:tasks.rados.rados.0.smithi059.stdout:72: done (0 left) 2024-04-01T13:30:47.553 INFO:tasks.rados.rados.0.smithi059.stdout:74: done (0 left) 2024-04-01T13:30:47.553 INFO:tasks.rados.rados.0.smithi059.stdout:75: write oid 45 current snap is 2 2024-04-01T13:30:47.553 INFO:tasks.rados.rados.0.smithi059.stdout:75: seq_num 61 ranges {625227=721013,1952931=704669} 2024-04-01T13:30:47.560 INFO:tasks.rados.rados.0.smithi059.stdout:75: writing smithi05931183-45 from 625227 to 1346240 tid 1 2024-04-01T13:30:47.568 INFO:tasks.rados.rados.0.smithi059.stdout:75: writing smithi05931183-45 from 1952931 to 2657600 tid 2 2024-04-01T13:30:47.569 INFO:tasks.rados.rados.0.smithi059.stdout:76: read oid 16 snap -1 2024-04-01T13:30:47.569 INFO:tasks.rados.rados.0.smithi059.stdout:76: expect (ObjNum 15 snap 0 seq_num 15) 2024-04-01T13:30:47.569 INFO:tasks.rados.rados.0.smithi059.stdout:77: write oid 46 current snap is 2 2024-04-01T13:30:47.569 INFO:tasks.rados.rados.0.smithi059.stdout:77: seq_num 62 ranges {642034=796262,1895060=564680} 2024-04-01T13:30:47.577 INFO:tasks.rados.rados.0.smithi059.stdout:77: writing smithi05931183-46 from 642034 to 1438296 tid 1 2024-04-01T13:30:47.583 INFO:tasks.rados.rados.0.smithi059.stdout:77: writing smithi05931183-46 from 1895060 to 2459740 tid 2 2024-04-01T13:30:47.584 INFO:tasks.rados.rados.0.smithi059.stdout:78: read oid 26 snap 0 2024-04-01T13:30:47.584 INFO:tasks.rados.rados.0.smithi059.stdout:78: expect (ObjNum 54 snap 0 seq_num 54) 2024-04-01T13:30:47.584 INFO:tasks.rados.rados.0.smithi059.stdout:79: read oid 7 snap -1 2024-04-01T13:30:47.584 INFO:tasks.rados.rados.0.smithi059.stdout:79: expect (ObjNum 6 snap 0 seq_num 6) 2024-04-01T13:30:47.584 INFO:tasks.rados.rados.0.smithi059.stdout:80: write oid 44 current snap is 2 2024-04-01T13:30:47.584 INFO:tasks.rados.rados.0.smithi059.stdout:80: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-04-01T13:30:47.591 INFO:tasks.rados.rados.0.smithi059.stdout:80: writing smithi05931183-44 from 658841 to 1130352 tid 1 2024-04-01T13:30:47.601 INFO:tasks.rados.rados.0.smithi059.stdout:80: writing smithi05931183-44 from 1553542 to 2261880 tid 2 2024-04-01T13:30:47.606 INFO:tasks.rados.rados.0.smithi059.stdout:80: writing smithi05931183-44 from 2964119 to 3058841 tid 3 2024-04-01T13:30:47.606 INFO:tasks.rados.rados.0.smithi059.stdout:81: read oid 30 snap -1 2024-04-01T13:30:47.606 INFO:tasks.rados.rados.0.smithi059.stdout:81: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-01T13:30:47.606 INFO:tasks.rados.rados.0.smithi059.stdout:82: snap_remove snap 1 2024-04-01T13:30:47.892 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T13:30:48.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:48.016+0000 7fb4ff55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:48.022+0000 7fb4ff55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:48.555 INFO:tasks.rados.rados.0.smithi059.stdout:82: done (7 left) 2024-04-01T13:30:48.555 INFO:tasks.rados.rados.0.smithi059.stdout:83: delete oid 49 current snap is 2 2024-04-01T13:30:48.555 INFO:tasks.rados.rados.0.smithi059.stdout:80: finishing write tid 1 to smithi05931183-44 2024-04-01T13:30:48.555 INFO:tasks.rados.rados.0.smithi059.stdout:76: expect (ObjNum 15 snap 0 seq_num 15) 2024-04-01T13:30:48.643 INFO:tasks.rados.rados.0.smithi059.stdout:79: expect (ObjNum 6 snap 0 seq_num 6) 2024-04-01T13:30:48.726 INFO:tasks.rados.rados.0.smithi059.stdout:81: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-01T13:30:48.824 INFO:tasks.rados.rados.0.smithi059.stdout:78: expect (ObjNum 54 snap 0 seq_num 54) 2024-04-01T13:30:48.976 INFO:tasks.rados.rados.0.smithi059.stdout:77: finishing write tid 1 to smithi05931183-46 2024-04-01T13:30:48.976 INFO:tasks.rados.rados.0.smithi059.stdout:75: finishing write tid 1 to smithi05931183-45 2024-04-01T13:30:48.976 INFO:tasks.rados.rados.0.smithi059.stdout:80: finishing write tid 2 to smithi05931183-44 2024-04-01T13:30:48.976 INFO:tasks.rados.rados.0.smithi059.stdout:77: finishing write tid 2 to smithi05931183-46 2024-04-01T13:30:48.977 INFO:tasks.rados.rados.0.smithi059.stdout:80: finishing write tid 3 to smithi05931183-44 2024-04-01T13:30:48.977 INFO:tasks.rados.rados.0.smithi059.stdout:75: finishing write tid 2 to smithi05931183-45 2024-04-01T13:30:48.977 INFO:tasks.rados.rados.0.smithi059.stdout:75: finishing write tid 3 to smithi05931183-45 2024-04-01T13:30:48.977 INFO:tasks.rados.rados.0.smithi059.stdout:77: finishing write tid 3 to smithi05931183-46 2024-04-01T13:30:48.977 INFO:tasks.rados.rados.0.smithi059.stdout:75: finishing write tid 4 to smithi05931183-45 2024-04-01T13:30:48.977 INFO:tasks.rados.rados.0.smithi059.stdout:75: oid 45 updating version 0 to 16 2024-04-01T13:30:48.977 INFO:tasks.rados.rados.0.smithi059.stdout:75: oid 45 updating version 16 to 18 2024-04-01T13:30:48.977 INFO:tasks.rados.rados.0.smithi059.stdout:75: oid 45 version 18 is already newer than 18 2024-04-01T13:30:48.977 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 45 v 18 (ObjNum 61 snap 2 seq_num 61) dirty exists 2024-04-01T13:30:48.977 INFO:tasks.rados.rados.0.smithi059.stdout:75: left oid 45 (ObjNum 61 snap 2 seq_num 61) 2024-04-01T13:30:48.977 INFO:tasks.rados.rados.0.smithi059.stdout:77: finishing write tid 4 to smithi05931183-46 2024-04-01T13:30:48.977 INFO:tasks.rados.rados.0.smithi059.stdout:77: oid 46 updating version 0 to 6 2024-04-01T13:30:48.977 INFO:tasks.rados.rados.0.smithi059.stdout:77: oid 46 version 6 is already newer than 6 2024-04-01T13:30:48.977 INFO:tasks.rados.rados.0.smithi059.stdout:77: oid 46 updating version 6 to 7 2024-04-01T13:30:48.978 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 46 v 7 (ObjNum 62 snap 2 seq_num 62) dirty exists 2024-04-01T13:30:48.978 INFO:tasks.rados.rados.0.smithi059.stdout:77: left oid 46 (ObjNum 62 snap 2 seq_num 62) 2024-04-01T13:30:48.978 INFO:tasks.rados.rados.0.smithi059.stdout:80: finishing write tid 4 to smithi05931183-44 2024-04-01T13:30:48.978 INFO:tasks.rados.rados.0.smithi059.stdout:75: done (7 left) 2024-04-01T13:30:48.978 INFO:tasks.rados.rados.0.smithi059.stdout:76: done (6 left) 2024-04-01T13:30:48.978 INFO:tasks.rados.rados.0.smithi059.stdout:77: done (5 left) 2024-04-01T13:30:48.978 INFO:tasks.rados.rados.0.smithi059.stdout:78: done (4 left) 2024-04-01T13:30:48.978 INFO:tasks.rados.rados.0.smithi059.stdout:79: done (3 left) 2024-04-01T13:30:48.978 INFO:tasks.rados.rados.0.smithi059.stdout:81: done (2 left) 2024-04-01T13:30:48.978 INFO:tasks.rados.rados.0.smithi059.stdout:83: done (1 left) 2024-04-01T13:30:48.978 INFO:tasks.rados.rados.0.smithi059.stdout:84: read oid 1 snap -1 2024-04-01T13:30:48.978 INFO:tasks.rados.rados.0.smithi059.stdout:84: expect deleted 2024-04-01T13:30:48.978 INFO:tasks.rados.rados.0.smithi059.stdout:85: read oid 47 snap 0 2024-04-01T13:30:48.978 INFO:tasks.rados.rados.0.smithi059.stdout:85: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-01T13:30:48.978 INFO:tasks.rados.rados.0.smithi059.stdout:86: write oid 10 current snap is 2 2024-04-01T13:30:48.978 INFO:tasks.rados.rados.0.smithi059.stdout:80: finishing write tid 5 to smithi05931183-44 2024-04-01T13:30:48.979 INFO:tasks.rados.rados.0.smithi059.stdout:80: oid 44 updating version 0 to 24 2024-04-01T13:30:48.979 INFO:tasks.rados.rados.0.smithi059.stdout:80: oid 44 updating version 24 to 27 2024-04-01T13:30:48.979 INFO:tasks.rados.rados.0.smithi059.stdout:80: oid 44 version 27 is already newer than 26 2024-04-01T13:30:48.979 INFO:tasks.rados.rados.0.smithi059.stdout:80: oid 44 version 27 is already newer than 25 2024-04-01T13:30:48.979 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 44 v 27 (ObjNum 63 snap 2 seq_num 63) dirty exists 2024-04-01T13:30:48.979 INFO:tasks.rados.rados.0.smithi059.stdout:80: left oid 44 (ObjNum 63 snap 2 seq_num 63) 2024-04-01T13:30:48.979 INFO:tasks.rados.rados.0.smithi059.stdout:86: seq_num 65 ranges {692455=622009,2070506=712007} 2024-04-01T13:30:48.983 INFO:tasks.rados.rados.0.smithi059.stdout:86: writing smithi05931183-10 from 692455 to 1314464 tid 1 2024-04-01T13:30:48.991 INFO:tasks.rados.rados.0.smithi059.stdout:86: writing smithi05931183-10 from 2070506 to 2782513 tid 2 2024-04-01T13:30:48.991 INFO:tasks.rados.rados.0.smithi059.stdout:80: done (3 left) 2024-04-01T13:30:48.991 INFO:tasks.rados.rados.0.smithi059.stdout:87: snap_create 2024-04-01T13:30:48.992 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 3 2024-04-01T13:30:49.026 INFO:tasks.rados.rados.0.smithi059.stdout:84: done (2 left) 2024-04-01T13:30:49.026 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 2 2024-04-01T13:30:49.040 INFO:tasks.rados.rados.0.smithi059.stdout:85: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-01T13:30:49.156 INFO:tasks.rados.rados.0.smithi059.stdout:86: finishing write tid 1 to smithi05931183-10 2024-04-01T13:30:49.157 INFO:tasks.rados.rados.0.smithi059.stdout:86: finishing write tid 2 to smithi05931183-10 2024-04-01T13:30:49.157 INFO:tasks.rados.rados.0.smithi059.stdout:86: finishing write tid 3 to smithi05931183-10 2024-04-01T13:30:49.157 INFO:tasks.rados.rados.0.smithi059.stdout:86: finishing write tid 4 to smithi05931183-10 2024-04-01T13:30:49.157 INFO:tasks.rados.rados.0.smithi059.stdout:86: oid 10 updating version 0 to 7 2024-04-01T13:30:49.157 INFO:tasks.rados.rados.0.smithi059.stdout:86: oid 10 version 7 is already newer than 5 2024-04-01T13:30:49.157 INFO:tasks.rados.rados.0.smithi059.stdout:86: oid 10 version 7 is already newer than 6 2024-04-01T13:30:49.157 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 10 v 7 (ObjNum 65 snap 2 seq_num 65) dirty exists 2024-04-01T13:30:49.157 INFO:tasks.rados.rados.0.smithi059.stdout:86: left oid 10 (ObjNum 65 snap 2 seq_num 65) 2024-04-01T13:30:49.157 INFO:tasks.rados.rados.0.smithi059.stdout:85: done (1 left) 2024-04-01T13:30:49.157 INFO:tasks.rados.rados.0.smithi059.stdout:86: done (0 left) 2024-04-01T13:30:49.568 INFO:tasks.rados.rados.0.smithi059.stdout:87: done (0 left) 2024-04-01T13:30:49.568 INFO:tasks.rados.rados.0.smithi059.stdout:88: write oid 6 current snap is 3 2024-04-01T13:30:49.568 INFO:tasks.rados.rados.0.smithi059.stdout:88: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2024-04-01T13:30:49.575 INFO:tasks.rados.rados.0.smithi059.stdout:88: writing smithi05931183-6 from 709262 to 1406520 tid 1 2024-04-01T13:30:49.581 INFO:tasks.rados.rados.0.smithi059.stdout:88: writing smithi05931183-6 from 2012635 to 2584653 tid 2 2024-04-01T13:30:49.585 INFO:tasks.rados.rados.0.smithi059.stdout:88: writing smithi05931183-6 from 3046388 to 3109262 tid 3 2024-04-01T13:30:49.585 INFO:tasks.rados.rados.0.smithi059.stdout:89: write oid 7 current snap is 3 2024-04-01T13:30:49.585 INFO:tasks.rados.rados.0.smithi059.stdout:89: seq_num 67 ranges {726069=772507,2071117=715676} 2024-04-01T13:30:49.593 INFO:tasks.rados.rados.0.smithi059.stdout:89: writing smithi05931183-7 from 726069 to 1498576 tid 1 2024-04-01T13:30:49.600 INFO:tasks.rados.rados.0.smithi059.stdout:89: writing smithi05931183-7 from 2071117 to 2786793 tid 2 2024-04-01T13:30:49.601 INFO:tasks.rados.rados.0.smithi059.stdout:90: write oid 22 current snap is 3 2024-04-01T13:30:49.601 INFO:tasks.rados.rados.0.smithi059.stdout:90: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2024-04-01T13:30:49.610 INFO:tasks.rados.rados.0.smithi059.stdout:90: writing smithi05931183-22 from 742876 to 1190632 tid 1 2024-04-01T13:30:49.614 INFO:tasks.rados.rados.0.smithi059.stdout:90: writing smithi05931183-22 from 1729599 to 2305286 tid 2 2024-04-01T13:30:49.619 INFO:tasks.rados.rados.0.smithi059.stdout:90: writing smithi05931183-22 from 3101234 to 3142876 tid 3 2024-04-01T13:30:49.619 INFO:tasks.rados.rados.0.smithi059.stdout:91: delete oid 14 current snap is 3 2024-04-01T13:30:49.649 INFO:tasks.rados.rados.0.smithi059.stdout:88: finishing write tid 1 to smithi05931183-6 2024-04-01T13:30:49.706 INFO:tasks.rados.rados.0.smithi059.stdout:91: done (3 left) 2024-04-01T13:30:49.706 INFO:tasks.rados.rados.0.smithi059.stdout:92: write (excl) oid 45 current snap is 3 2024-04-01T13:30:49.706 INFO:tasks.rados.rados.0.smithi059.stdout:92: seq_num 70 ranges {776490=714607,2246563=579356} 2024-04-01T13:30:49.708 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-01T13:30:49.708 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-01T13:30:49.708 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2, 3] 2024-04-01T13:30:49.709 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-01T13:30:49.709 INFO:teuthology.orchestra.run:waiting for 300 2024-04-01T13:30:49.714 INFO:tasks.rados.rados.0.smithi059.stdout:92: writing smithi05931183-45 from 776490 to 1491097 tid 1 2024-04-01T13:30:49.723 INFO:tasks.rados.rados.0.smithi059.stdout:92: writing smithi05931183-45 from 2246563 to 2825919 tid 2 2024-04-01T13:30:49.723 INFO:tasks.rados.rados.0.smithi059.stdout:93: write oid 27 current snap is 3 2024-04-01T13:30:49.723 INFO:tasks.rados.rados.0.smithi059.stdout:93: seq_num 71 ranges {793297=789856,2305045=723014} 2024-04-01T13:30:49.731 INFO:tasks.rados.rados.0.smithi059.stdout:93: writing smithi05931183-27 from 793297 to 1583153 tid 1 2024-04-01T13:30:49.741 INFO:tasks.rados.rados.0.smithi059.stdout:93: writing smithi05931183-27 from 2305045 to 3028059 tid 2 2024-04-01T13:30:49.741 INFO:tasks.rados.rados.0.smithi059.stdout:94: read oid 14 snap -1 2024-04-01T13:30:49.741 INFO:tasks.rados.rados.0.smithi059.stdout:94: expect deleted 2024-04-01T13:30:49.741 INFO:tasks.rados.rados.0.smithi059.stdout:95: read oid 4 snap -1 2024-04-01T13:30:49.741 INFO:tasks.rados.rados.0.smithi059.stdout:95: expect (ObjNum 56 snap 0 seq_num 56) 2024-04-01T13:30:49.741 INFO:tasks.rados.rados.0.smithi059.stdout:96: delete oid 49 current snap is 3 2024-04-01T13:30:49.741 INFO:tasks.rados.rados.0.smithi059.stdout:90: finishing write tid 1 to smithi05931183-22 2024-04-01T13:30:49.741 INFO:tasks.rados.rados.0.smithi059.stdout:88: finishing write tid 2 to smithi05931183-6 2024-04-01T13:30:49.746 INFO:tasks.rados.rados.0.smithi059.stdout:96: done (7 left) 2024-04-01T13:30:49.746 INFO:tasks.rados.rados.0.smithi059.stdout:97: snap_remove snap 0 2024-04-01T13:30:49.781 INFO:tasks.ceph.osd.0:Stopped 2024-04-01T13:30:50.108 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T13:30:50.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:50.232+0000 7fb0f4f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:50.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:50.238+0000 7fb0f4f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:50.573 INFO:tasks.rados.rados.0.smithi059.stdout:97: done (7 left) 2024-04-01T13:30:50.573 INFO:tasks.rados.rados.0.smithi059.stdout:98: write (excl) oid 34 current snap is 3 2024-04-01T13:30:50.573 INFO:tasks.rados.rados.0.smithi059.stdout:98: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2024-04-01T13:30:50.578 INFO:tasks.rados.rados.0.smithi059.stdout:98: writing smithi05931183-34 from 426911 to 967265 tid 1 2024-04-01T13:30:50.586 INFO:tasks.rados.rados.0.smithi059.stdout:98: writing smithi05931183-34 from 1622009 to 2348692 tid 2 2024-04-01T13:30:50.591 INFO:tasks.rados.rados.0.smithi059.stdout:98: writing smithi05931183-34 from 3121996 to 3226911 tid 3 2024-04-01T13:30:50.591 INFO:tasks.rados.rados.0.smithi059.stdout:99: snap_remove snap 2 2024-04-01T13:30:51.580 INFO:tasks.rados.rados.0.smithi059.stdout:99: done (8 left) 2024-04-01T13:30:51.580 INFO:tasks.rados.rados.0.smithi059.stdout:100: write oid 21 current snap is 3 2024-04-01T13:30:51.580 INFO:tasks.rados.rados.0.smithi059.stdout:100: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2024-04-01T13:30:51.586 INFO:tasks.rados.rados.0.smithi059.stdout:100: writing smithi05931183-21 from 443718 to 1059321 tid 1 2024-04-01T13:30:51.591 INFO:tasks.rados.rados.0.smithi059.stdout:100: writing smithi05931183-21 from 1564138 to 2034479 tid 2 2024-04-01T13:30:51.599 INFO:tasks.rados.rados.0.smithi059.stdout:100: writing smithi05931183-21 from 2633066 to 3243718 tid 3 2024-04-01T13:30:51.599 INFO:tasks.rados.rados.0.smithi059.stdout:101: read oid 47 snap -1 2024-04-01T13:30:51.599 INFO:tasks.rados.rados.0.smithi059.stdout:101: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-01T13:30:51.600 INFO:tasks.rados.rados.0.smithi059.stdout:102: read oid 3 snap -1 2024-04-01T13:30:51.600 INFO:tasks.rados.rados.0.smithi059.stdout:102: expect (ObjNum 2 snap 0 seq_num 2) 2024-04-01T13:30:51.600 INFO:tasks.rados.rados.0.smithi059.stdout:93: finishing write tid 1 to smithi05931183-27 2024-04-01T13:30:51.600 INFO:tasks.rados.rados.0.smithi059.stdout:93: finishing write tid 2 to smithi05931183-27 2024-04-01T13:30:51.600 INFO:tasks.rados.rados.0.smithi059.stdout:93: finishing write tid 3 to smithi05931183-27 2024-04-01T13:30:51.600 INFO:tasks.rados.rados.0.smithi059.stdout:93: finishing write tid 4 to smithi05931183-27 2024-04-01T13:30:51.600 INFO:tasks.rados.rados.0.smithi059.stdout:93: oid 27 updating version 0 to 19 2024-04-01T13:30:51.600 INFO:tasks.rados.rados.0.smithi059.stdout:93: oid 27 version 19 is already newer than 18 2024-04-01T13:30:51.600 INFO:tasks.rados.rados.0.smithi059.stdout:93: oid 27 updating version 19 to 20 2024-04-01T13:30:51.600 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 27 v 20 (ObjNum 71 snap 3 seq_num 71) dirty exists 2024-04-01T13:30:51.600 INFO:tasks.rados.rados.0.smithi059.stdout:93: left oid 27 (ObjNum 71 snap 3 seq_num 71) 2024-04-01T13:30:51.600 INFO:tasks.rados.rados.0.smithi059.stdout:103: read oid 39 snap -1 2024-04-01T13:30:51.600 INFO:tasks.rados.rados.0.smithi059.stdout:103: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-01T13:30:51.600 INFO:tasks.rados.rados.0.smithi059.stdout:93: done (11 left) 2024-04-01T13:30:51.600 INFO:tasks.rados.rados.0.smithi059.stdout:94: done (10 left) 2024-04-01T13:30:51.601 INFO:tasks.rados.rados.0.smithi059.stdout:104: read oid 1 snap -1 2024-04-01T13:30:51.601 INFO:tasks.rados.rados.0.smithi059.stdout:104: expect deleted 2024-04-01T13:30:51.601 INFO:tasks.rados.rados.0.smithi059.stdout:105: read oid 13 snap -1 2024-04-01T13:30:51.601 INFO:tasks.rados.rados.0.smithi059.stdout:105: expect (ObjNum 55 snap 0 seq_num 55) 2024-04-01T13:30:51.601 INFO:tasks.rados.rados.0.smithi059.stdout:106: read oid 27 snap -1 2024-04-01T13:30:51.601 INFO:tasks.rados.rados.0.smithi059.stdout:106: expect (ObjNum 71 snap 3 seq_num 71) 2024-04-01T13:30:51.601 INFO:tasks.rados.rados.0.smithi059.stdout:107: delete oid 11 current snap is 3 2024-04-01T13:30:51.601 INFO:tasks.rados.rados.0.smithi059.stdout:95: expect (ObjNum 56 snap 0 seq_num 56) 2024-04-01T13:30:51.759 INFO:tasks.rados.rados.0.smithi059.stdout:106: expect (ObjNum 71 snap 3 seq_num 71) 2024-04-01T13:30:54.781 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3] 2024-04-01T13:30:54.782 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-01T13:30:54.782 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-01T13:30:54.782 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-01T13:30:54.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:54.918+0000 7f26e2a64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:30:54.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:30:54.928+0000 7f26e2a64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:20.081 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-01T13:31:20.082 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-01T13:31:20.085 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-01T13:31:20.086 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-01T13:31:20.086 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-01T13:31:20.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:20.227+0000 7f2c0eb54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:20.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:20.236+0000 7f2c0eb54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:20.239 INFO:teuthology.orchestra.run.smithi059.stderr:no valid command found; 10 closest matches: 2024-04-01T13:31:20.240 INFO:teuthology.orchestra.run.smithi059.stderr:config get 2024-04-01T13:31:20.240 INFO:teuthology.orchestra.run.smithi059.stderr:config help 2024-04-01T13:31:20.240 INFO:teuthology.orchestra.run.smithi059.stderr:config set ... 2024-04-01T13:31:20.240 INFO:teuthology.orchestra.run.smithi059.stderr:config show 2024-04-01T13:31:20.240 INFO:teuthology.orchestra.run.smithi059.stderr:deep_scrub 2024-04-01T13:31:20.240 INFO:teuthology.orchestra.run.smithi059.stderr:dump_historic_ops 2024-04-01T13:31:20.240 INFO:teuthology.orchestra.run.smithi059.stderr:dump_historic_slow_ops 2024-04-01T13:31:20.240 INFO:teuthology.orchestra.run.smithi059.stderr:dump_metrics [] 2024-04-01T13:31:20.240 INFO:teuthology.orchestra.run.smithi059.stderr:dump_ops_in_flight 2024-04-01T13:31:20.240 INFO:teuthology.orchestra.run.smithi059.stderr:dump_pgstate_history 2024-04-01T13:31:20.240 INFO:teuthology.orchestra.run.smithi059.stderr:Error EINVAL: invalid command 2024-04-01T13:31:20.243 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:31:20.243 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-01T13:31:20.243 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-01T13:31:20.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:20.367+0000 7f8f0e4ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:20.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:20.376+0000 7f8f0e4ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:20.379 INFO:teuthology.orchestra.run.smithi059.stderr:no valid command found; 10 closest matches: 2024-04-01T13:31:20.379 INFO:teuthology.orchestra.run.smithi059.stderr:config get 2024-04-01T13:31:20.379 INFO:teuthology.orchestra.run.smithi059.stderr:config help 2024-04-01T13:31:20.379 INFO:teuthology.orchestra.run.smithi059.stderr:config set ... 2024-04-01T13:31:20.379 INFO:teuthology.orchestra.run.smithi059.stderr:config show 2024-04-01T13:31:20.379 INFO:teuthology.orchestra.run.smithi059.stderr:deep_scrub 2024-04-01T13:31:20.379 INFO:teuthology.orchestra.run.smithi059.stderr:dump_historic_ops 2024-04-01T13:31:20.379 INFO:teuthology.orchestra.run.smithi059.stderr:dump_historic_slow_ops 2024-04-01T13:31:20.379 INFO:teuthology.orchestra.run.smithi059.stderr:dump_metrics [] 2024-04-01T13:31:20.379 INFO:teuthology.orchestra.run.smithi059.stderr:dump_ops_in_flight 2024-04-01T13:31:20.379 INFO:teuthology.orchestra.run.smithi059.stderr:dump_pgstate_history 2024-04-01T13:31:20.379 INFO:teuthology.orchestra.run.smithi059.stderr:Error EINVAL: invalid command 2024-04-01T13:31:20.382 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:31:20.382 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-01T13:31:20.382 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-01T13:31:20.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:20.504+0000 7f4f9fdfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:20.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:20.514+0000 7f4f9fdfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:20.518 INFO:teuthology.orchestra.run.smithi059.stderr:no valid command found; 10 closest matches: 2024-04-01T13:31:20.518 INFO:teuthology.orchestra.run.smithi059.stderr:config get 2024-04-01T13:31:20.518 INFO:teuthology.orchestra.run.smithi059.stderr:config help 2024-04-01T13:31:20.518 INFO:teuthology.orchestra.run.smithi059.stderr:config set ... 2024-04-01T13:31:20.518 INFO:teuthology.orchestra.run.smithi059.stderr:config show 2024-04-01T13:31:20.518 INFO:teuthology.orchestra.run.smithi059.stderr:deep_scrub 2024-04-01T13:31:20.518 INFO:teuthology.orchestra.run.smithi059.stderr:dump_historic_ops 2024-04-01T13:31:20.518 INFO:teuthology.orchestra.run.smithi059.stderr:dump_historic_slow_ops 2024-04-01T13:31:20.518 INFO:teuthology.orchestra.run.smithi059.stderr:dump_metrics [] 2024-04-01T13:31:20.518 INFO:teuthology.orchestra.run.smithi059.stderr:dump_ops_in_flight 2024-04-01T13:31:20.518 INFO:teuthology.orchestra.run.smithi059.stderr:dump_pgstate_history 2024-04-01T13:31:20.519 INFO:teuthology.orchestra.run.smithi059.stderr:Error EINVAL: invalid command 2024-04-01T13:31:20.521 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:31:20.521 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-01T13:31:21.207 INFO:tasks.rados.rados.0.smithi059.stdout:98: finishing write tid 1 to smithi05931183-34 2024-04-01T13:31:21.231 INFO:tasks.rados.rados.0.smithi059.stdout:98: finishing write tid 2 to smithi05931183-34 2024-04-01T13:31:21.239 INFO:tasks.rados.rados.0.smithi059.stdout:98: finishing write tid 3 to smithi05931183-34 2024-04-01T13:31:21.242 INFO:tasks.rados.rados.0.smithi059.stdout:98: finishing write tid 4 to smithi05931183-34 2024-04-01T13:31:21.244 INFO:tasks.rados.rados.0.smithi059.stdout:98: finishing write tid 5 to smithi05931183-34 2024-04-01T13:31:21.244 INFO:tasks.rados.rados.0.smithi059.stdout:98: oid 34 updating version 0 to 25 2024-04-01T13:31:21.244 INFO:tasks.rados.rados.0.smithi059.stdout:98: oid 34 version 25 is already newer than 25 2024-04-01T13:31:21.244 INFO:tasks.rados.rados.0.smithi059.stdout:98: oid 34 version 25 is already newer than 22 2024-04-01T13:31:21.244 INFO:tasks.rados.rados.0.smithi059.stdout:98: oid 34 version 25 is already newer than 23 2024-04-01T13:31:21.244 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 34 v 25 (ObjNum 73 snap 3 seq_num 73) dirty exists 2024-04-01T13:31:21.244 INFO:tasks.rados.rados.0.smithi059.stdout:98: left oid 34 (ObjNum 73 snap 3 seq_num 73) 2024-04-01T13:31:21.272 INFO:tasks.rados.rados.0.smithi059.stdout:103: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-01T13:31:21.375 INFO:tasks.rados.rados.0.smithi059.stdout:95: done (13 left) 2024-04-01T13:31:21.375 INFO:tasks.rados.rados.0.smithi059.stdout:98: done (12 left) 2024-04-01T13:31:21.375 INFO:tasks.rados.rados.0.smithi059.stdout:103: done (11 left) 2024-04-01T13:31:21.375 INFO:tasks.rados.rados.0.smithi059.stdout:106: done (10 left) 2024-04-01T13:31:21.375 INFO:tasks.rados.rados.0.smithi059.stdout:107: done (9 left) 2024-04-01T13:31:21.375 INFO:tasks.rados.rados.0.smithi059.stdout:108: snap_create 2024-04-01T13:31:21.375 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 9 2024-04-01T13:31:25.522 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3] 2024-04-01T13:31:25.522 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-01T13:31:25.523 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-04-01T13:31:25.523 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-01T13:31:25.523 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-01T13:31:25.526 INFO:tasks.ceph.osd.0:Started 2024-04-01T13:31:25.526 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-01T13:31:25.639 INFO:teuthology.orchestra.run.smithi059.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:31:25.642 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:31:25.642 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-01T13:31:25.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-01T13:31:30.644 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-01T13:31:30.759 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-01T13:31:30.759 INFO:teuthology.orchestra.run.smithi059.stdout: "ops_in_flight": [ 2024-04-01T13:31:30.759 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-01T13:31:30.759 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client_request", 2024-04-01T13:31:30.759 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 19, 2024-04-01T13:31:30.759 INFO:teuthology.orchestra.run.smithi059.stdout: "detail": { 2024-04-01T13:31:30.759 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.759 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:31:30.759 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.511800+0000" 2024-04-01T13:31:30.759 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.760 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.760 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:31:30.760 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.511815+0000" 2024-04-01T13:31:30.760 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.760 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.760 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:31:30.760 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.518190+0000" 2024-04-01T13:31:30.760 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.760 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.760 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:31:30.760 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.760 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.760 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.761 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:31:30.761 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.741749+0000" 2024-04-01T13:31:30.761 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.761 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.761 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:31:30.761 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.744576+0000" 2024-04-01T13:31:30.761 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.761 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.761 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:31:30.761 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.761 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.761 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.761 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:31:30.761 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.762 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.762 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.762 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.762 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-01T13:31:30.762 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client_request", 2024-04-01T13:31:30.762 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 20, 2024-04-01T13:31:30.762 INFO:teuthology.orchestra.run.smithi059.stdout: "detail": { 2024-04-01T13:31:30.762 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.762 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:31:30.762 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.513107+0000" 2024-04-01T13:31:30.762 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.762 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.762 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:31:30.763 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.513116+0000" 2024-04-01T13:31:30.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.763 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.763 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:31:30.763 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.518285+0000" 2024-04-01T13:31:30.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.763 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.763 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:31:30.763 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.763 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.763 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:31:30.763 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.741971+0000" 2024-04-01T13:31:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:31:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.747053+0000" 2024-04-01T13:31:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:31:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:31:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.765 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.765 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-01T13:31:30.765 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client_request", 2024-04-01T13:31:30.765 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 21, 2024-04-01T13:31:30.765 INFO:teuthology.orchestra.run.smithi059.stdout: "detail": { 2024-04-01T13:31:30.765 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.765 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:31:30.765 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.513430+0000" 2024-04-01T13:31:30.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.765 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.765 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:31:30.765 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.513439+0000" 2024-04-01T13:31:30.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.766 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.766 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:31:30.766 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.518387+0000" 2024-04-01T13:31:30.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.766 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.766 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:31:30.766 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.766 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.766 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:31:30.766 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.742346+0000" 2024-04-01T13:31:30.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.766 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.766 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:31:30.767 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.747523+0000" 2024-04-01T13:31:30.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.767 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.767 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:31:30.767 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.767 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.767 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:31:30.767 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.767 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.767 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-01T13:31:30.767 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client_request", 2024-04-01T13:31:30.767 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 22, 2024-04-01T13:31:30.768 INFO:teuthology.orchestra.run.smithi059.stdout: "detail": { 2024-04-01T13:31:30.768 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.768 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:31:30.768 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.513690+0000" 2024-04-01T13:31:30.768 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.768 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.768 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:31:30.768 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.513698+0000" 2024-04-01T13:31:30.768 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.768 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.768 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:31:30.768 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.518483+0000" 2024-04-01T13:31:30.768 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.768 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.768 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:31:30.769 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.769 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.769 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:31:30.769 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.743627+0000" 2024-04-01T13:31:30.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.769 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.769 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:31:30.769 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.749420+0000" 2024-04-01T13:31:30.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.769 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.769 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:31:30.769 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.769 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.769 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:31:30.770 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.770 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.770 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.770 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.770 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-01T13:31:30.770 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client_request", 2024-04-01T13:31:30.770 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 26, 2024-04-01T13:31:30.770 INFO:teuthology.orchestra.run.smithi059.stdout: "detail": { 2024-04-01T13:31:30.770 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.770 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:31:30.770 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.515258+0000" 2024-04-01T13:31:30.770 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.770 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.770 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:31:30.771 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.515267+0000" 2024-04-01T13:31:30.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.771 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.771 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:31:30.771 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.518851+0000" 2024-04-01T13:31:30.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.771 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.771 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:31:30.771 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.771 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.771 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:31:30.771 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.746939+0000" 2024-04-01T13:31:30.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.771 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.772 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:31:30.772 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.757892+0000" 2024-04-01T13:31:30.772 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.772 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.772 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:31:30.772 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.772 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.772 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.772 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:31:30.772 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.772 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.772 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.772 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.772 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-01T13:31:30.772 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client_request", 2024-04-01T13:31:30.773 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 27, 2024-04-01T13:31:30.773 INFO:teuthology.orchestra.run.smithi059.stdout: "detail": { 2024-04-01T13:31:30.773 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.773 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:31:30.773 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.515570+0000" 2024-04-01T13:31:30.773 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.773 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.773 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:31:30.773 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.515579+0000" 2024-04-01T13:31:30.773 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.773 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.773 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:31:30.773 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.519008+0000" 2024-04-01T13:31:30.773 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.774 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.774 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:31:30.774 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.774 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.774 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:31:30.774 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.748716+0000" 2024-04-01T13:31:30.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.774 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.774 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:31:30.774 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.758679+0000" 2024-04-01T13:31:30.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.774 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.774 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:31:30.775 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.775 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.775 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.775 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:31:30.775 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.775 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.775 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.775 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.775 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-01T13:31:30.775 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client_request", 2024-04-01T13:31:30.775 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 28, 2024-04-01T13:31:30.775 INFO:teuthology.orchestra.run.smithi059.stdout: "detail": { 2024-04-01T13:31:30.775 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.775 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:31:30.775 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.515996+0000" 2024-04-01T13:31:30.775 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.776 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.776 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:31:30.776 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.516005+0000" 2024-04-01T13:31:30.776 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.776 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.776 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:31:30.776 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.519239+0000" 2024-04-01T13:31:30.776 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.776 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.776 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:31:30.776 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.776 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.776 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.776 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:31:30.776 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.750970+0000" 2024-04-01T13:31:30.777 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.777 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.777 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:31:30.777 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.759037+0000" 2024-04-01T13:31:30.777 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.777 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.777 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:31:30.777 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.777 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.777 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.777 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:31:30.777 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.777 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.777 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.777 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.777 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-01T13:31:30.777 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client_request", 2024-04-01T13:31:30.778 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 29, 2024-04-01T13:31:30.778 INFO:teuthology.orchestra.run.smithi059.stdout: "detail": { 2024-04-01T13:31:30.778 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.778 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:31:30.778 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.516324+0000" 2024-04-01T13:31:30.778 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.778 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.778 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:31:30.778 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.516333+0000" 2024-04-01T13:31:30.778 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.778 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.778 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:31:30.778 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.519421+0000" 2024-04-01T13:31:30.778 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.779 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.779 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:31:30.779 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.779 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.779 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:31:30.779 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.754437+0000" 2024-04-01T13:31:30.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.779 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.779 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:31:30.779 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.759414+0000" 2024-04-01T13:31:30.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.779 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.779 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:31:30.779 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.780 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.780 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.780 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:31:30.780 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.780 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.780 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.780 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.780 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-01T13:31:30.780 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client_request", 2024-04-01T13:31:30.780 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 30, 2024-04-01T13:31:30.780 INFO:teuthology.orchestra.run.smithi059.stdout: "detail": { 2024-04-01T13:31:30.780 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.780 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:31:30.780 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.516585+0000" 2024-04-01T13:31:30.780 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.780 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.781 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:31:30.781 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.516593+0000" 2024-04-01T13:31:30.781 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.781 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.781 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:31:30.781 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.519525+0000" 2024-04-01T13:31:30.781 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.781 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.781 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:31:30.781 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.781 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.781 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.781 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:31:30.781 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.756911+0000" 2024-04-01T13:31:30.781 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.781 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.781 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:31:30.782 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.759793+0000" 2024-04-01T13:31:30.782 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.782 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.782 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:31:30.782 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.782 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.782 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.782 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:31:30.782 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.782 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.782 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.782 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.782 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-01T13:31:30.782 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client_request", 2024-04-01T13:31:30.782 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 31, 2024-04-01T13:31:30.783 INFO:teuthology.orchestra.run.smithi059.stdout: "detail": { 2024-04-01T13:31:30.783 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.783 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:31:30.783 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.516923+0000" 2024-04-01T13:31:30.783 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.783 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.783 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:31:30.783 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.516932+0000" 2024-04-01T13:31:30.783 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.783 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.783 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:31:30.783 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.519624+0000" 2024-04-01T13:31:30.783 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.783 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.783 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:31:30.783 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.784 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.784 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.784 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:31:30.784 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.757956+0000" 2024-04-01T13:31:30.784 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.784 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.784 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:31:30.784 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.760601+0000" 2024-04-01T13:31:30.784 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.784 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.784 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:31:30.784 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.784 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.784 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.784 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:31:30.784 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.785 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.785 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.785 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.785 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-01T13:31:30.785 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client_request", 2024-04-01T13:31:30.785 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 15, 2024-04-01T13:31:30.785 INFO:teuthology.orchestra.run.smithi059.stdout: "detail": { 2024-04-01T13:31:30.785 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.785 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:31:30.785 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.501903+0000" 2024-04-01T13:31:30.785 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.785 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.785 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:31:30.785 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.501946+0000" 2024-04-01T13:31:30.785 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.786 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.786 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:31:30.786 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.517297+0000" 2024-04-01T13:31:30.786 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.786 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.786 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:31:30.786 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.517732+0000" 2024-04-01T13:31:30.786 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.786 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.786 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:31:30.786 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.740301+0000" 2024-04-01T13:31:30.786 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.786 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.786 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:31:30.787 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.741660+0000" 2024-04-01T13:31:30.787 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.787 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.787 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:31:30.787 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.787 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.787 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.787 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:31:30.787 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.787 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.787 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.787 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.787 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-01T13:31:30.787 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client_request", 2024-04-01T13:31:30.788 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 16, 2024-04-01T13:31:30.788 INFO:teuthology.orchestra.run.smithi059.stdout: "detail": { 2024-04-01T13:31:30.788 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.788 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:31:30.788 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.505108+0000" 2024-04-01T13:31:30.788 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.788 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.788 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:31:30.788 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.505135+0000" 2024-04-01T13:31:30.788 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.788 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.788 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:31:30.788 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.517464+0000" 2024-04-01T13:31:30.788 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.789 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.789 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:31:30.789 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.789 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.789 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.789 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:31:30.789 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.740767+0000" 2024-04-01T13:31:30.789 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.789 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.789 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:31:30.789 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.742671+0000" 2024-04-01T13:31:30.789 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.789 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.789 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:31:30.789 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.789 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.790 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.790 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:31:30.790 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.790 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.790 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.790 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.790 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-01T13:31:30.790 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client_request", 2024-04-01T13:31:30.790 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 18, 2024-04-01T13:31:30.790 INFO:teuthology.orchestra.run.smithi059.stdout: "detail": { 2024-04-01T13:31:30.790 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.790 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:31:30.790 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.507808+0000" 2024-04-01T13:31:30.790 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.791 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.791 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:31:30.791 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.507817+0000" 2024-04-01T13:31:30.791 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.791 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.791 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:31:30.791 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.518076+0000" 2024-04-01T13:31:30.791 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.791 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.791 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:31:30.791 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.791 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.791 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.791 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:31:30.792 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.741469+0000" 2024-04-01T13:31:30.792 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.792 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.792 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:31:30.792 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.744443+0000" 2024-04-01T13:31:30.792 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.792 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.792 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:31:30.792 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.792 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.792 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.792 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:31:30.792 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.793 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.793 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.793 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.793 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-01T13:31:30.793 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client_request", 2024-04-01T13:31:30.793 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 17, 2024-04-01T13:31:30.793 INFO:teuthology.orchestra.run.smithi059.stdout: "detail": { 2024-04-01T13:31:30.793 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.793 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:31:30.793 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.507546+0000" 2024-04-01T13:31:30.793 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.793 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.793 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:31:30.793 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.507557+0000" 2024-04-01T13:31:30.794 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.794 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.794 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:31:30.794 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.517944+0000" 2024-04-01T13:31:30.794 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.794 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.794 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:31:30.794 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.794 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.794 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.794 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:31:30.794 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.741145+0000" 2024-04-01T13:31:30.794 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.794 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.795 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:31:30.795 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.744371+0000" 2024-04-01T13:31:30.795 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.795 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.795 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:31:30.795 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.795 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.795 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.795 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:31:30.795 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.795 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.795 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.795 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.795 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-01T13:31:30.795 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client_request", 2024-04-01T13:31:30.796 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 23, 2024-04-01T13:31:30.796 INFO:teuthology.orchestra.run.smithi059.stdout: "detail": { 2024-04-01T13:31:30.796 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.796 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:31:30.796 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.514150+0000" 2024-04-01T13:31:30.796 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.796 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.796 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:31:30.796 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.514159+0000" 2024-04-01T13:31:30.796 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.796 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.796 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:31:30.796 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.518575+0000" 2024-04-01T13:31:30.797 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.797 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.797 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:31:30.797 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.797 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.797 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.797 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:31:30.797 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.743979+0000" 2024-04-01T13:31:30.797 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.797 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.797 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:31:30.797 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.755387+0000" 2024-04-01T13:31:30.797 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.797 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.798 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:31:30.798 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.798 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.798 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.798 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:31:30.798 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.798 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.798 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.798 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.798 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-01T13:31:30.798 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client_request", 2024-04-01T13:31:30.798 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 24, 2024-04-01T13:31:30.798 INFO:teuthology.orchestra.run.smithi059.stdout: "detail": { 2024-04-01T13:31:30.799 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.799 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:31:30.799 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.514489+0000" 2024-04-01T13:31:30.799 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.799 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.799 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:31:30.799 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.514498+0000" 2024-04-01T13:31:30.799 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.799 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.799 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:31:30.799 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.518667+0000" 2024-04-01T13:31:30.799 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.799 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.800 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:31:30.800 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.800 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.800 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.800 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:31:30.800 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.745390+0000" 2024-04-01T13:31:30.800 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.800 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.800 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:31:30.800 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.757888+0000" 2024-04-01T13:31:30.800 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.800 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.800 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:31:30.800 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.800 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.800 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.800 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:31:30.801 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.801 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.801 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.801 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.801 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-01T13:31:30.801 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client_request", 2024-04-01T13:31:30.801 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 25, 2024-04-01T13:31:30.801 INFO:teuthology.orchestra.run.smithi059.stdout: "detail": { 2024-04-01T13:31:30.801 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.801 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:31:30.801 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.514926+0000" 2024-04-01T13:31:30.801 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.801 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.801 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:31:30.801 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.514935+0000" 2024-04-01T13:31:30.802 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.802 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.802 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:31:30.802 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.518760+0000" 2024-04-01T13:31:30.802 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.802 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.802 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:31:30.802 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.802 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.802 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.802 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:31:30.802 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.745773+0000" 2024-04-01T13:31:30.802 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.802 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.803 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:31:30.803 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-01T13:31:29.758915+0000" 2024-04-01T13:31:30.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.803 INFO:teuthology.orchestra.run.smithi059.stdout: "blocking_event": { 2024-04-01T13:31:30.803 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:31:30.803 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-01T13:31:30.803 INFO:teuthology.orchestra.run.smithi059.stdout: "time_event": { 2024-04-01T13:31:30.803 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:31:30.803 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "0.000000" 2024-04-01T13:31:30.803 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.803 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.803 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-01T13:31:30.803 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-01T13:31:30.803 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-01T13:31:31.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2024-04-01T13:31:35.805 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 0] 2024-04-01T13:31:35.805 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-01T13:31:35.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:35.943+0000 7f1524d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:35.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:35.954+0000 7f1524d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:36.541 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.0 to 1 (10000) 2024-04-01T13:31:36.555 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-01T13:31:36.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:36.683+0000 7fd4d415d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:36.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:36.694+0000 7fd4d415d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:36.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~11s 2024-04-01T13:31:37.547 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.1 to 1 (10000) 2024-04-01T13:31:37.564 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-01T13:31:37.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:37.697+0000 7f77a7420640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:37.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:37.708+0000 7f77a7420640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:38.549 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.2 to 1 (10000) 2024-04-01T13:31:38.566 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-01T13:31:38.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:38.708+0000 7fac9b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:38.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:38.718+0000 7fac9b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:39.553 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.3 to 1 (10000) 2024-04-01T13:31:39.570 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-01T13:31:39.570 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T13:31:39.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:39.702+0000 7f59fd86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:39.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:39.710+0000 7f59fd86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:39.890 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:31:39.890 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-01T13:31:39.902 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-04-01T13:31:39.568999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99162169,"num_objects":44,"num_object_clones":4,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":286,"num_read_kb":73310,"num_write":376,"num_write_kb":67988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":337,"ondisk_log_size":337,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":284800,"kb_used_data":174672,"kb_used_omap":1,"kb_used_meta":107582,"kb_avail":377202560,"statfs":{"total":386547056640,"available":386255421440,"internally_reserved":0,"allocated":178864128,"data_stored":183025549,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":1508,"internal_metadata":110164508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.187262"},"pg_stats":[{"pgid":"3.f","version":"16'9","reported_seq":43,"reported_epoch":30,"state":"peering","last_fresh":"2024-04-01T13:31:29.607744+0000","last_change":"2024-04-01T13:31:29.607744+0000","last_active":"2024-04-01T13:31:29.607744+0000","last_peered":"2024-04-01T13:30:45.695648+0000","last_clean":"2024-04-01T13:30:45.695648+0000","last_became_active":"2024-04-01T13:30:45.686209+0000","last_became_peered":"2024-04-01T13:30:45.686209+0000","last_unstale":"2024-04-01T13:31:29.607744+0000","last_undegraded":"2024-04-01T13:31:29.607744+0000","last_fullsized":"2024-04-01T13:31:29.607744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7008420,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":4853,"num_write":12,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"26'11","reported_seq":64,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.534360+0000","last_change":"2024-04-01T13:30:53.709220+0000","last_active":"2024-04-01T13:31:30.534360+0000","last_peered":"2024-04-01T13:31:30.534360+0000","last_clean":"2024-04-01T13:31:30.534360+0000","last_became_active":"2024-04-01T13:30:45.689067+0000","last_became_peered":"2024-04-01T13:30:45.689067+0000","last_unstale":"2024-04-01T13:31:30.534360+0000","last_undegraded":"2024-04-01T13:31:30.534360+0000","last_fullsized":"2024-04-01T13:31:30.534360+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042034,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"25'23","reported_seq":81,"reported_epoch":26,"state":"active+clean+snaptrim","last_fresh":"2024-04-01T13:30:51.638760+0000","last_change":"2024-04-01T13:30:50.616859+0000","last_active":"2024-04-01T13:30:51.638760+0000","last_peered":"2024-04-01T13:30:51.638760+0000","last_clean":"2024-04-01T13:30:51.638760+0000","last_became_active":"2024-04-01T13:30:45.692470+0000","last_became_peered":"2024-04-01T13:30:45.692470+0000","last_unstale":"2024-04-01T13:30:51.638760+0000","last_undegraded":"2024-04-01T13:30:51.638760+0000","last_fullsized":"2024-04-01T13:30:51.638760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":10722734,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":5638,"num_write":26,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"28'26","reported_seq":71,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.696045+0000","last_change":"2024-04-01T13:31:30.697568+0000","last_active":"2024-04-01T13:31:30.696045+0000","last_peered":"2024-04-01T13:31:30.696045+0000","last_clean":"2024-04-01T13:30:29.519276+0000","last_became_active":"2024-04-01T13:31:30.574937+0000","last_became_peered":"2024-04-01T13:31:30.574937+0000","last_unstale":"2024-04-01T13:31:30.696045+0000","last_undegraded":"2024-04-01T13:31:30.696045+0000","last_fullsized":"2024-04-01T13:31:30.696045+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"16'12","reported_seq":86,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.627295+0000","last_change":"2024-04-01T13:31:36.596629+0000","last_active":"2024-04-01T13:31:38.627295+0000","last_peered":"2024-04-01T13:31:38.627295+0000","last_clean":"2024-04-01T13:31:38.627295+0000","last_became_active":"2024-04-01T13:31:30.604705+0000","last_became_peered":"2024-04-01T13:31:30.604705+0000","last_unstale":"2024-04-01T13:31:38.627295+0000","last_undegraded":"2024-04-01T13:31:38.627295+0000","last_fullsized":"2024-04-01T13:31:38.627295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":10843,"num_write":16,"num_write_kb":4752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"24'11","reported_seq":56,"reported_epoch":30,"state":"peering","last_fresh":"2024-04-01T13:31:29.600971+0000","last_change":"2024-04-01T13:31:29.600971+0000","last_active":"2024-04-01T13:31:29.600971+0000","last_peered":"2024-04-01T13:30:49.664233+0000","last_clean":"2024-04-01T13:30:49.664233+0000","last_became_active":"2024-04-01T13:30:30.656628+0000","last_became_peered":"2024-04-01T13:30:30.656628+0000","last_unstale":"2024-04-01T13:31:29.600971+0000","last_undegraded":"2024-04-01T13:31:29.600971+0000","last_fullsized":"2024-04-01T13:31:29.600971+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7495823,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10286,"num_write":13,"num_write_kb":3933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'20","reported_seq":56,"reported_epoch":30,"state":"peering","last_fresh":"2024-04-01T13:31:29.542796+0000","last_change":"2024-04-01T13:31:29.542796+0000","last_active":"2024-04-01T13:31:29.542796+0000","last_peered":"2024-04-01T13:30:49.030107+0000","last_clean":"2024-04-01T13:30:49.030107+0000","last_became_active":"2024-04-01T13:30:30.635917+0000","last_became_peered":"2024-04-01T13:30:30.635917+0000","last_unstale":"2024-04-01T13:31:29.542796+0000","last_undegraded":"2024-04-01T13:31:29.542796+0000","last_fullsized":"2024-04-01T13:31:29.542796+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5865577,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":22,"num_write_kb":6047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"24'20","reported_seq":78,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.697588+0000","last_change":"2024-04-01T13:30:51.647095+0000","last_active":"2024-04-01T13:30:51.697588+0000","last_peered":"2024-04-01T13:30:51.697588+0000","last_clean":"2024-04-01T13:30:51.697588+0000","last_became_active":"2024-04-01T13:30:45.695964+0000","last_became_peered":"2024-04-01T13:30:45.695964+0000","last_unstale":"2024-04-01T13:30:51.697588+0000","last_undegraded":"2024-04-01T13:30:51.697588+0000","last_fullsized":"2024-04-01T13:30:51.697588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11243751,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":14764,"num_write":26,"num_write_kb":7239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":186,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.621853+0000","last_change":"2024-04-01T13:31:30.585158+0000","last_active":"2024-04-01T13:31:38.621853+0000","last_peered":"2024-04-01T13:31:38.621853+0000","last_clean":"2024-04-01T13:31:38.621853+0000","last_became_active":"2024-04-01T13:31:30.578315+0000","last_became_peered":"2024-04-01T13:31:30.578315+0000","last_unstale":"2024-04-01T13:31:38.621853+0000","last_undegraded":"2024-04-01T13:31:38.621853+0000","last_fullsized":"2024-04-01T13:31:38.621853+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_clean_scrub_stamp":"2024-04-01T13:30:22.378617+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.643211+0000","last_change":"2024-04-01T13:30:24.493976+0000","last_active":"2024-04-01T13:30:51.643211+0000","last_peered":"2024-04-01T13:30:51.643211+0000","last_clean":"2024-04-01T13:30:51.643211+0000","last_became_active":"2024-04-01T13:30:24.483528+0000","last_became_peered":"2024-04-01T13:30:24.483528+0000","last_unstale":"2024-04-01T13:30:51.643211+0000","last_undegraded":"2024-04-01T13:30:51.643211+0000","last_fullsized":"2024-04-01T13:30:51.643211+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"24'6","reported_seq":50,"reported_epoch":26,"state":"active+clean+snaptrim","last_fresh":"2024-04-01T13:30:51.640989+0000","last_change":"2024-04-01T13:30:50.619384+0000","last_active":"2024-04-01T13:30:51.640989+0000","last_peered":"2024-04-01T13:30:51.640989+0000","last_clean":"2024-04-01T13:30:51.640989+0000","last_became_active":"2024-04-01T13:30:45.703077+0000","last_became_peered":"2024-04-01T13:30:45.703077+0000","last_unstale":"2024-04-01T13:30:51.640989+0000","last_undegraded":"2024-04-01T13:30:51.640989+0000","last_fullsized":"2024-04-01T13:30:51.640989+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2352947,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"peering","last_fresh":"2024-04-01T13:31:29.731657+0000","last_change":"2024-04-01T13:31:29.731657+0000","last_active":"2024-04-01T13:31:29.731657+0000","last_peered":"2024-04-01T13:30:24.494919+0000","last_clean":"2024-04-01T13:30:24.494919+0000","last_became_active":"2024-04-01T13:30:24.486637+0000","last_became_peered":"2024-04-01T13:30:24.486637+0000","last_unstale":"2024-04-01T13:31:29.731657+0000","last_undegraded":"2024-04-01T13:31:29.731657+0000","last_fullsized":"2024-04-01T13:31:29.731657+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"26'13","reported_seq":67,"reported_epoch":33,"state":"peering","last_fresh":"2024-04-01T13:31:37.629763+0000","last_change":"2024-04-01T13:31:29.552816+0000","last_active":"2024-04-01T13:31:20.179135+0000","last_peered":"2024-04-01T13:30:51.678933+0000","last_clean":"2024-04-01T13:30:51.678933+0000","last_became_active":"2024-04-01T13:30:45.719645+0000","last_became_peered":"2024-04-01T13:30:45.719645+0000","last_unstale":"2024-04-01T13:31:37.629763+0000","last_undegraded":"2024-04-01T13:31:37.629763+0000","last_fullsized":"2024-04-01T13:31:37.629763+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8050454,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10129,"num_write":17,"num_write_kb":4892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":52,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.629011+0000","last_change":"2024-04-01T13:30:45.679818+0000","last_active":"2024-04-01T13:30:51.629011+0000","last_peered":"2024-04-01T13:30:51.629011+0000","last_clean":"2024-04-01T13:30:51.629011+0000","last_became_active":"2024-04-01T13:30:45.674014+0000","last_became_peered":"2024-04-01T13:30:45.674014+0000","last_unstale":"2024-04-01T13:30:51.629011+0000","last_undegraded":"2024-04-01T13:30:51.629011+0000","last_fullsized":"2024-04-01T13:30:51.629011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"26'35","reported_seq":100,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.619593+0000","last_change":"2024-04-01T13:30:53.716341+0000","last_active":"2024-04-01T13:31:38.619593+0000","last_peered":"2024-04-01T13:31:38.619593+0000","last_clean":"2024-04-01T13:31:38.619593+0000","last_became_active":"2024-04-01T13:30:45.733223+0000","last_became_peered":"2024-04-01T13:30:45.733223+0000","last_unstale":"2024-04-01T13:31:38.619593+0000","last_undegraded":"2024-04-01T13:31:38.619593+0000","last_fullsized":"2024-04-01T13:31:38.619593+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":12168103,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":5339,"num_write":35,"num_write_kb":8781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"peering","last_fresh":"2024-04-01T13:31:29.726707+0000","last_change":"2024-04-01T13:31:29.726707+0000","last_active":"2024-04-01T13:31:29.726707+0000","last_peered":"2024-04-01T13:30:24.480593+0000","last_clean":"2024-04-01T13:30:24.480593+0000","last_became_active":"2024-04-01T13:30:24.472751+0000","last_became_peered":"2024-04-01T13:30:24.472751+0000","last_unstale":"2024-04-01T13:31:29.726707+0000","last_undegraded":"2024-04-01T13:31:29.726707+0000","last_fullsized":"2024-04-01T13:31:29.726707+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"26'9","reported_seq":76,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.689469+0000","last_change":"2024-04-01T13:31:30.690519+0000","last_active":"2024-04-01T13:31:30.689469+0000","last_peered":"2024-04-01T13:31:30.689469+0000","last_clean":"2024-04-01T13:30:51.628188+0000","last_became_active":"2024-04-01T13:31:30.573263+0000","last_became_peered":"2024-04-01T13:31:30.573263+0000","last_unstale":"2024-04-01T13:31:30.689469+0000","last_undegraded":"2024-04-01T13:31:30.689469+0000","last_fullsized":"2024-04-01T13:31:30.689469+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"26'15","reported_seq":61,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.656955+0000","last_change":"2024-04-01T13:30:51.714578+0000","last_active":"2024-04-01T13:30:51.656955+0000","last_peered":"2024-04-01T13:30:51.656955+0000","last_clean":"2024-04-01T13:30:51.656955+0000","last_became_active":"2024-04-01T13:30:45.690525+0000","last_became_peered":"2024-04-01T13:30:45.690525+0000","last_unstale":"2024-04-01T13:30:51.656955+0000","last_undegraded":"2024-04-01T13:30:51.656955+0000","last_fullsized":"2024-04-01T13:30:51.656955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067228,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":13,"num_write_kb":2827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.583299+0000","last_change":"2024-04-01T13:31:30.584389+0000","last_active":"2024-04-01T13:31:30.583299+0000","last_peered":"2024-04-01T13:31:30.583299+0000","last_clean":"2024-04-01T13:30:51.632893+0000","last_became_active":"2024-04-01T13:31:30.579930+0000","last_became_peered":"2024-04-01T13:31:30.579930+0000","last_unstale":"2024-04-01T13:31:30.583299+0000","last_undegraded":"2024-04-01T13:31:30.583299+0000","last_fullsized":"2024-04-01T13:31:30.583299+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"16'6","reported_seq":69,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.590018+0000","last_change":"2024-04-01T13:31:30.591443+0000","last_active":"2024-04-01T13:31:30.590018+0000","last_peered":"2024-04-01T13:31:30.590018+0000","last_clean":"2024-04-01T13:30:51.638376+0000","last_became_active":"2024-04-01T13:31:30.581362+0000","last_became_peered":"2024-04-01T13:31:30.581362+0000","last_unstale":"2024-04-01T13:31:30.590018+0000","last_undegraded":"2024-04-01T13:31:30.590018+0000","last_fullsized":"2024-04-01T13:31:30.590018+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4907578,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5282,"num_write":8,"num_write_kb":2191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":56,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.627006+0000","last_change":"2024-04-01T13:31:30.608391+0000","last_active":"2024-04-01T13:31:38.627006+0000","last_peered":"2024-04-01T13:31:38.627006+0000","last_clean":"2024-04-01T13:31:38.627006+0000","last_became_active":"2024-04-01T13:31:30.601821+0000","last_became_peered":"2024-04-01T13:31:30.601821+0000","last_unstale":"2024-04-01T13:31:38.627006+0000","last_undegraded":"2024-04-01T13:31:38.627006+0000","last_fullsized":"2024-04-01T13:31:38.627006+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"24'24","reported_seq":75,"reported_epoch":33,"state":"peering+snaptrim","last_fresh":"2024-04-01T13:31:37.623842+0000","last_change":"2024-04-01T13:31:29.513840+0000","last_active":"2024-04-01T13:31:20.116167+0000","last_peered":"2024-04-01T13:30:51.627795+0000","last_clean":"2024-04-01T13:30:51.627795+0000","last_became_active":"2024-04-01T13:30:45.693544+0000","last_became_peered":"2024-04-01T13:30:45.693544+0000","last_unstale":"2024-04-01T13:31:37.623842+0000","last_undegraded":"2024-04-01T13:31:37.623842+0000","last_fullsized":"2024-04-01T13:31:37.623842+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739541,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":6072,"num_write":29,"num_write_kb":8248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.593772+0000","last_change":"2024-04-01T13:31:30.594953+0000","last_active":"2024-04-01T13:31:30.593772+0000","last_peered":"2024-04-01T13:31:30.593772+0000","last_clean":"2024-04-01T13:30:51.631427+0000","last_became_active":"2024-04-01T13:31:30.586687+0000","last_became_peered":"2024-04-01T13:31:30.586687+0000","last_unstale":"2024-04-01T13:31:30.593772+0000","last_undegraded":"2024-04-01T13:31:30.593772+0000","last_fullsized":"2024-04-01T13:31:30.593772+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.532772+0000","last_change":"2024-04-01T13:30:51.650292+0000","last_active":"2024-04-01T13:31:30.532772+0000","last_peered":"2024-04-01T13:31:30.532772+0000","last_clean":"2024-04-01T13:31:30.532772+0000","last_became_active":"2024-04-01T13:30:45.693590+0000","last_became_peered":"2024-04-01T13:30:45.693590+0000","last_unstale":"2024-04-01T13:31:30.532772+0000","last_undegraded":"2024-04-01T13:31:30.532772+0000","last_fullsized":"2024-04-01T13:31:30.532772+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.643299+0000","last_change":"2024-04-01T13:30:24.495962+0000","last_active":"2024-04-01T13:30:51.643299+0000","last_peered":"2024-04-01T13:30:51.643299+0000","last_clean":"2024-04-01T13:30:51.643299+0000","last_became_active":"2024-04-01T13:30:24.489422+0000","last_became_peered":"2024-04-01T13:30:24.489422+0000","last_unstale":"2024-04-01T13:30:51.643299+0000","last_undegraded":"2024-04-01T13:30:51.643299+0000","last_fullsized":"2024-04-01T13:30:51.643299+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":98571801,"num_objects":42,"num_object_clones":4,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":183,"num_read_kb":73219,"num_write":239,"num_write_kb":66094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76812,"kb_used_data":49796,"kb_used_omap":0,"kb_used_meta":27008,"kb_avail":94295028,"statfs":{"total":96636764160,"available":96558108672,"internally_reserved":0,"allocated":50991104,"data_stored":51408199,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27656192},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64556,"kb_used_data":37732,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94307284,"statfs":{"total":96636764160,"available":96570658816,"internally_reserved":0,"allocated":38637568,"data_stored":39479250,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69828,"kb_used_data":42448,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94302012,"statfs":{"total":96636764160,"available":96565260288,"internally_reserved":0,"allocated":43466752,"data_stored":44221816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":29,"seq":124554051589,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73604,"kb_used_data":44696,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94298236,"statfs":{"total":96636764160,"available":96561393664,"internally_reserved":0,"allocated":45768704,"data_stored":47916284,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1508,"internal_metadata":27589148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-01T13:31:39.903 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T13:31:40.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:40.025+0000 7fb7778ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:40.033+0000 7fb7778ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:40.214 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:31:40.215 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-01T13:31:40.227 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-04-01T13:31:39.568999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99162169,"num_objects":44,"num_object_clones":4,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":286,"num_read_kb":73310,"num_write":376,"num_write_kb":67988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":337,"ondisk_log_size":337,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":284800,"kb_used_data":174672,"kb_used_omap":1,"kb_used_meta":107582,"kb_avail":377202560,"statfs":{"total":386547056640,"available":386255421440,"internally_reserved":0,"allocated":178864128,"data_stored":183025549,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":1508,"internal_metadata":110164508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.187262"},"pg_stats":[{"pgid":"3.f","version":"16'9","reported_seq":43,"reported_epoch":30,"state":"peering","last_fresh":"2024-04-01T13:31:29.607744+0000","last_change":"2024-04-01T13:31:29.607744+0000","last_active":"2024-04-01T13:31:29.607744+0000","last_peered":"2024-04-01T13:30:45.695648+0000","last_clean":"2024-04-01T13:30:45.695648+0000","last_became_active":"2024-04-01T13:30:45.686209+0000","last_became_peered":"2024-04-01T13:30:45.686209+0000","last_unstale":"2024-04-01T13:31:29.607744+0000","last_undegraded":"2024-04-01T13:31:29.607744+0000","last_fullsized":"2024-04-01T13:31:29.607744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7008420,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":4853,"num_write":12,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"26'11","reported_seq":64,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.534360+0000","last_change":"2024-04-01T13:30:53.709220+0000","last_active":"2024-04-01T13:31:30.534360+0000","last_peered":"2024-04-01T13:31:30.534360+0000","last_clean":"2024-04-01T13:31:30.534360+0000","last_became_active":"2024-04-01T13:30:45.689067+0000","last_became_peered":"2024-04-01T13:30:45.689067+0000","last_unstale":"2024-04-01T13:31:30.534360+0000","last_undegraded":"2024-04-01T13:31:30.534360+0000","last_fullsized":"2024-04-01T13:31:30.534360+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042034,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"25'23","reported_seq":81,"reported_epoch":26,"state":"active+clean+snaptrim","last_fresh":"2024-04-01T13:30:51.638760+0000","last_change":"2024-04-01T13:30:50.616859+0000","last_active":"2024-04-01T13:30:51.638760+0000","last_peered":"2024-04-01T13:30:51.638760+0000","last_clean":"2024-04-01T13:30:51.638760+0000","last_became_active":"2024-04-01T13:30:45.692470+0000","last_became_peered":"2024-04-01T13:30:45.692470+0000","last_unstale":"2024-04-01T13:30:51.638760+0000","last_undegraded":"2024-04-01T13:30:51.638760+0000","last_fullsized":"2024-04-01T13:30:51.638760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":10722734,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":5638,"num_write":26,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"28'26","reported_seq":71,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.696045+0000","last_change":"2024-04-01T13:31:30.697568+0000","last_active":"2024-04-01T13:31:30.696045+0000","last_peered":"2024-04-01T13:31:30.696045+0000","last_clean":"2024-04-01T13:30:29.519276+0000","last_became_active":"2024-04-01T13:31:30.574937+0000","last_became_peered":"2024-04-01T13:31:30.574937+0000","last_unstale":"2024-04-01T13:31:30.696045+0000","last_undegraded":"2024-04-01T13:31:30.696045+0000","last_fullsized":"2024-04-01T13:31:30.696045+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"16'12","reported_seq":86,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.627295+0000","last_change":"2024-04-01T13:31:36.596629+0000","last_active":"2024-04-01T13:31:38.627295+0000","last_peered":"2024-04-01T13:31:38.627295+0000","last_clean":"2024-04-01T13:31:38.627295+0000","last_became_active":"2024-04-01T13:31:30.604705+0000","last_became_peered":"2024-04-01T13:31:30.604705+0000","last_unstale":"2024-04-01T13:31:38.627295+0000","last_undegraded":"2024-04-01T13:31:38.627295+0000","last_fullsized":"2024-04-01T13:31:38.627295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":10843,"num_write":16,"num_write_kb":4752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"24'11","reported_seq":56,"reported_epoch":30,"state":"peering","last_fresh":"2024-04-01T13:31:29.600971+0000","last_change":"2024-04-01T13:31:29.600971+0000","last_active":"2024-04-01T13:31:29.600971+0000","last_peered":"2024-04-01T13:30:49.664233+0000","last_clean":"2024-04-01T13:30:49.664233+0000","last_became_active":"2024-04-01T13:30:30.656628+0000","last_became_peered":"2024-04-01T13:30:30.656628+0000","last_unstale":"2024-04-01T13:31:29.600971+0000","last_undegraded":"2024-04-01T13:31:29.600971+0000","last_fullsized":"2024-04-01T13:31:29.600971+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7495823,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10286,"num_write":13,"num_write_kb":3933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'20","reported_seq":56,"reported_epoch":30,"state":"peering","last_fresh":"2024-04-01T13:31:29.542796+0000","last_change":"2024-04-01T13:31:29.542796+0000","last_active":"2024-04-01T13:31:29.542796+0000","last_peered":"2024-04-01T13:30:49.030107+0000","last_clean":"2024-04-01T13:30:49.030107+0000","last_became_active":"2024-04-01T13:30:30.635917+0000","last_became_peered":"2024-04-01T13:30:30.635917+0000","last_unstale":"2024-04-01T13:31:29.542796+0000","last_undegraded":"2024-04-01T13:31:29.542796+0000","last_fullsized":"2024-04-01T13:31:29.542796+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5865577,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":22,"num_write_kb":6047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"24'20","reported_seq":78,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.697588+0000","last_change":"2024-04-01T13:30:51.647095+0000","last_active":"2024-04-01T13:30:51.697588+0000","last_peered":"2024-04-01T13:30:51.697588+0000","last_clean":"2024-04-01T13:30:51.697588+0000","last_became_active":"2024-04-01T13:30:45.695964+0000","last_became_peered":"2024-04-01T13:30:45.695964+0000","last_unstale":"2024-04-01T13:30:51.697588+0000","last_undegraded":"2024-04-01T13:30:51.697588+0000","last_fullsized":"2024-04-01T13:30:51.697588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11243751,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":14764,"num_write":26,"num_write_kb":7239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":186,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.621853+0000","last_change":"2024-04-01T13:31:30.585158+0000","last_active":"2024-04-01T13:31:38.621853+0000","last_peered":"2024-04-01T13:31:38.621853+0000","last_clean":"2024-04-01T13:31:38.621853+0000","last_became_active":"2024-04-01T13:31:30.578315+0000","last_became_peered":"2024-04-01T13:31:30.578315+0000","last_unstale":"2024-04-01T13:31:38.621853+0000","last_undegraded":"2024-04-01T13:31:38.621853+0000","last_fullsized":"2024-04-01T13:31:38.621853+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_clean_scrub_stamp":"2024-04-01T13:30:22.378617+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.643211+0000","last_change":"2024-04-01T13:30:24.493976+0000","last_active":"2024-04-01T13:30:51.643211+0000","last_peered":"2024-04-01T13:30:51.643211+0000","last_clean":"2024-04-01T13:30:51.643211+0000","last_became_active":"2024-04-01T13:30:24.483528+0000","last_became_peered":"2024-04-01T13:30:24.483528+0000","last_unstale":"2024-04-01T13:30:51.643211+0000","last_undegraded":"2024-04-01T13:30:51.643211+0000","last_fullsized":"2024-04-01T13:30:51.643211+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"24'6","reported_seq":50,"reported_epoch":26,"state":"active+clean+snaptrim","last_fresh":"2024-04-01T13:30:51.640989+0000","last_change":"2024-04-01T13:30:50.619384+0000","last_active":"2024-04-01T13:30:51.640989+0000","last_peered":"2024-04-01T13:30:51.640989+0000","last_clean":"2024-04-01T13:30:51.640989+0000","last_became_active":"2024-04-01T13:30:45.703077+0000","last_became_peered":"2024-04-01T13:30:45.703077+0000","last_unstale":"2024-04-01T13:30:51.640989+0000","last_undegraded":"2024-04-01T13:30:51.640989+0000","last_fullsized":"2024-04-01T13:30:51.640989+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2352947,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"peering","last_fresh":"2024-04-01T13:31:29.731657+0000","last_change":"2024-04-01T13:31:29.731657+0000","last_active":"2024-04-01T13:31:29.731657+0000","last_peered":"2024-04-01T13:30:24.494919+0000","last_clean":"2024-04-01T13:30:24.494919+0000","last_became_active":"2024-04-01T13:30:24.486637+0000","last_became_peered":"2024-04-01T13:30:24.486637+0000","last_unstale":"2024-04-01T13:31:29.731657+0000","last_undegraded":"2024-04-01T13:31:29.731657+0000","last_fullsized":"2024-04-01T13:31:29.731657+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"26'13","reported_seq":67,"reported_epoch":33,"state":"peering","last_fresh":"2024-04-01T13:31:37.629763+0000","last_change":"2024-04-01T13:31:29.552816+0000","last_active":"2024-04-01T13:31:20.179135+0000","last_peered":"2024-04-01T13:30:51.678933+0000","last_clean":"2024-04-01T13:30:51.678933+0000","last_became_active":"2024-04-01T13:30:45.719645+0000","last_became_peered":"2024-04-01T13:30:45.719645+0000","last_unstale":"2024-04-01T13:31:37.629763+0000","last_undegraded":"2024-04-01T13:31:37.629763+0000","last_fullsized":"2024-04-01T13:31:37.629763+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8050454,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10129,"num_write":17,"num_write_kb":4892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":52,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.629011+0000","last_change":"2024-04-01T13:30:45.679818+0000","last_active":"2024-04-01T13:30:51.629011+0000","last_peered":"2024-04-01T13:30:51.629011+0000","last_clean":"2024-04-01T13:30:51.629011+0000","last_became_active":"2024-04-01T13:30:45.674014+0000","last_became_peered":"2024-04-01T13:30:45.674014+0000","last_unstale":"2024-04-01T13:30:51.629011+0000","last_undegraded":"2024-04-01T13:30:51.629011+0000","last_fullsized":"2024-04-01T13:30:51.629011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"26'35","reported_seq":100,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.619593+0000","last_change":"2024-04-01T13:30:53.716341+0000","last_active":"2024-04-01T13:31:38.619593+0000","last_peered":"2024-04-01T13:31:38.619593+0000","last_clean":"2024-04-01T13:31:38.619593+0000","last_became_active":"2024-04-01T13:30:45.733223+0000","last_became_peered":"2024-04-01T13:30:45.733223+0000","last_unstale":"2024-04-01T13:31:38.619593+0000","last_undegraded":"2024-04-01T13:31:38.619593+0000","last_fullsized":"2024-04-01T13:31:38.619593+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":12168103,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":5339,"num_write":35,"num_write_kb":8781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"peering","last_fresh":"2024-04-01T13:31:29.726707+0000","last_change":"2024-04-01T13:31:29.726707+0000","last_active":"2024-04-01T13:31:29.726707+0000","last_peered":"2024-04-01T13:30:24.480593+0000","last_clean":"2024-04-01T13:30:24.480593+0000","last_became_active":"2024-04-01T13:30:24.472751+0000","last_became_peered":"2024-04-01T13:30:24.472751+0000","last_unstale":"2024-04-01T13:31:29.726707+0000","last_undegraded":"2024-04-01T13:31:29.726707+0000","last_fullsized":"2024-04-01T13:31:29.726707+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"26'9","reported_seq":76,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.689469+0000","last_change":"2024-04-01T13:31:30.690519+0000","last_active":"2024-04-01T13:31:30.689469+0000","last_peered":"2024-04-01T13:31:30.689469+0000","last_clean":"2024-04-01T13:30:51.628188+0000","last_became_active":"2024-04-01T13:31:30.573263+0000","last_became_peered":"2024-04-01T13:31:30.573263+0000","last_unstale":"2024-04-01T13:31:30.689469+0000","last_undegraded":"2024-04-01T13:31:30.689469+0000","last_fullsized":"2024-04-01T13:31:30.689469+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"26'15","reported_seq":61,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.656955+0000","last_change":"2024-04-01T13:30:51.714578+0000","last_active":"2024-04-01T13:30:51.656955+0000","last_peered":"2024-04-01T13:30:51.656955+0000","last_clean":"2024-04-01T13:30:51.656955+0000","last_became_active":"2024-04-01T13:30:45.690525+0000","last_became_peered":"2024-04-01T13:30:45.690525+0000","last_unstale":"2024-04-01T13:30:51.656955+0000","last_undegraded":"2024-04-01T13:30:51.656955+0000","last_fullsized":"2024-04-01T13:30:51.656955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067228,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":13,"num_write_kb":2827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.583299+0000","last_change":"2024-04-01T13:31:30.584389+0000","last_active":"2024-04-01T13:31:30.583299+0000","last_peered":"2024-04-01T13:31:30.583299+0000","last_clean":"2024-04-01T13:30:51.632893+0000","last_became_active":"2024-04-01T13:31:30.579930+0000","last_became_peered":"2024-04-01T13:31:30.579930+0000","last_unstale":"2024-04-01T13:31:30.583299+0000","last_undegraded":"2024-04-01T13:31:30.583299+0000","last_fullsized":"2024-04-01T13:31:30.583299+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"16'6","reported_seq":69,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.590018+0000","last_change":"2024-04-01T13:31:30.591443+0000","last_active":"2024-04-01T13:31:30.590018+0000","last_peered":"2024-04-01T13:31:30.590018+0000","last_clean":"2024-04-01T13:30:51.638376+0000","last_became_active":"2024-04-01T13:31:30.581362+0000","last_became_peered":"2024-04-01T13:31:30.581362+0000","last_unstale":"2024-04-01T13:31:30.590018+0000","last_undegraded":"2024-04-01T13:31:30.590018+0000","last_fullsized":"2024-04-01T13:31:30.590018+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4907578,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5282,"num_write":8,"num_write_kb":2191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":56,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.627006+0000","last_change":"2024-04-01T13:31:30.608391+0000","last_active":"2024-04-01T13:31:38.627006+0000","last_peered":"2024-04-01T13:31:38.627006+0000","last_clean":"2024-04-01T13:31:38.627006+0000","last_became_active":"2024-04-01T13:31:30.601821+0000","last_became_peered":"2024-04-01T13:31:30.601821+0000","last_unstale":"2024-04-01T13:31:38.627006+0000","last_undegraded":"2024-04-01T13:31:38.627006+0000","last_fullsized":"2024-04-01T13:31:38.627006+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"24'24","reported_seq":75,"reported_epoch":33,"state":"peering+snaptrim","last_fresh":"2024-04-01T13:31:37.623842+0000","last_change":"2024-04-01T13:31:29.513840+0000","last_active":"2024-04-01T13:31:20.116167+0000","last_peered":"2024-04-01T13:30:51.627795+0000","last_clean":"2024-04-01T13:30:51.627795+0000","last_became_active":"2024-04-01T13:30:45.693544+0000","last_became_peered":"2024-04-01T13:30:45.693544+0000","last_unstale":"2024-04-01T13:31:37.623842+0000","last_undegraded":"2024-04-01T13:31:37.623842+0000","last_fullsized":"2024-04-01T13:31:37.623842+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739541,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":6072,"num_write":29,"num_write_kb":8248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.593772+0000","last_change":"2024-04-01T13:31:30.594953+0000","last_active":"2024-04-01T13:31:30.593772+0000","last_peered":"2024-04-01T13:31:30.593772+0000","last_clean":"2024-04-01T13:30:51.631427+0000","last_became_active":"2024-04-01T13:31:30.586687+0000","last_became_peered":"2024-04-01T13:31:30.586687+0000","last_unstale":"2024-04-01T13:31:30.593772+0000","last_undegraded":"2024-04-01T13:31:30.593772+0000","last_fullsized":"2024-04-01T13:31:30.593772+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.532772+0000","last_change":"2024-04-01T13:30:51.650292+0000","last_active":"2024-04-01T13:31:30.532772+0000","last_peered":"2024-04-01T13:31:30.532772+0000","last_clean":"2024-04-01T13:31:30.532772+0000","last_became_active":"2024-04-01T13:30:45.693590+0000","last_became_peered":"2024-04-01T13:30:45.693590+0000","last_unstale":"2024-04-01T13:31:30.532772+0000","last_undegraded":"2024-04-01T13:31:30.532772+0000","last_fullsized":"2024-04-01T13:31:30.532772+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.643299+0000","last_change":"2024-04-01T13:30:24.495962+0000","last_active":"2024-04-01T13:30:51.643299+0000","last_peered":"2024-04-01T13:30:51.643299+0000","last_clean":"2024-04-01T13:30:51.643299+0000","last_became_active":"2024-04-01T13:30:24.489422+0000","last_became_peered":"2024-04-01T13:30:24.489422+0000","last_unstale":"2024-04-01T13:30:51.643299+0000","last_undegraded":"2024-04-01T13:30:51.643299+0000","last_fullsized":"2024-04-01T13:30:51.643299+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":98571801,"num_objects":42,"num_object_clones":4,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":183,"num_read_kb":73219,"num_write":239,"num_write_kb":66094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76812,"kb_used_data":49796,"kb_used_omap":0,"kb_used_meta":27008,"kb_avail":94295028,"statfs":{"total":96636764160,"available":96558108672,"internally_reserved":0,"allocated":50991104,"data_stored":51408199,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27656192},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64556,"kb_used_data":37732,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94307284,"statfs":{"total":96636764160,"available":96570658816,"internally_reserved":0,"allocated":38637568,"data_stored":39479250,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69828,"kb_used_data":42448,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94302012,"statfs":{"total":96636764160,"available":96565260288,"internally_reserved":0,"allocated":43466752,"data_stored":44221816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":29,"seq":124554051589,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73604,"kb_used_data":44696,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94298236,"statfs":{"total":96636764160,"available":96561393664,"internally_reserved":0,"allocated":45768704,"data_stored":47916284,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1508,"internal_metadata":27589148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-01T13:31:40.228 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-01T13:31:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:40.350+0000 7f0926cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:40.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:40.357+0000 7f0926cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:40.611 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:31:40.612 INFO:teuthology.orchestra.run.smithi059.stdout:{"fsid":"2a626e12-75d1-4fa9-b59b-fc64723c30fe","health":{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 2 pgs peering","count":2},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":87,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1711978289,"num_in_osds":4,"osd_in_since":1711978215,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"peering","count":6},{"state_name":"active+clean+snaptrim","count":2},{"state_name":"peering+snaptrim","count":1}],"num_pgs":25,"num_pools":3,"num_objects":44,"data_bytes":99162169,"bytes_used":291635200,"bytes_avail":386255421440,"bytes_total":386547056640,"inactive_pgs_ratio":0.2800000011920929,"recovering_objects_per_sec":0,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"read_bytes_sec":435431,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-01T13:31:19.377894+0000","services":{}},"progress_events":{}} 2024-04-01T13:31:40.624 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-01T13:31:40.624 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T13:31:40.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:40.747+0000 7fca76c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:40.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:40.755+0000 7fca76c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:40.942 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:31:40.942 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-01T13:31:40.954 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-04-01T13:31:39.568999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99162169,"num_objects":44,"num_object_clones":4,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":286,"num_read_kb":73310,"num_write":376,"num_write_kb":67988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":337,"ondisk_log_size":337,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":284800,"kb_used_data":174672,"kb_used_omap":1,"kb_used_meta":107582,"kb_avail":377202560,"statfs":{"total":386547056640,"available":386255421440,"internally_reserved":0,"allocated":178864128,"data_stored":183025549,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":1508,"internal_metadata":110164508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.187262"},"pg_stats":[{"pgid":"3.f","version":"16'9","reported_seq":43,"reported_epoch":30,"state":"peering","last_fresh":"2024-04-01T13:31:29.607744+0000","last_change":"2024-04-01T13:31:29.607744+0000","last_active":"2024-04-01T13:31:29.607744+0000","last_peered":"2024-04-01T13:30:45.695648+0000","last_clean":"2024-04-01T13:30:45.695648+0000","last_became_active":"2024-04-01T13:30:45.686209+0000","last_became_peered":"2024-04-01T13:30:45.686209+0000","last_unstale":"2024-04-01T13:31:29.607744+0000","last_undegraded":"2024-04-01T13:31:29.607744+0000","last_fullsized":"2024-04-01T13:31:29.607744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7008420,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":4853,"num_write":12,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"26'11","reported_seq":64,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.534360+0000","last_change":"2024-04-01T13:30:53.709220+0000","last_active":"2024-04-01T13:31:30.534360+0000","last_peered":"2024-04-01T13:31:30.534360+0000","last_clean":"2024-04-01T13:31:30.534360+0000","last_became_active":"2024-04-01T13:30:45.689067+0000","last_became_peered":"2024-04-01T13:30:45.689067+0000","last_unstale":"2024-04-01T13:31:30.534360+0000","last_undegraded":"2024-04-01T13:31:30.534360+0000","last_fullsized":"2024-04-01T13:31:30.534360+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042034,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"25'23","reported_seq":81,"reported_epoch":26,"state":"active+clean+snaptrim","last_fresh":"2024-04-01T13:30:51.638760+0000","last_change":"2024-04-01T13:30:50.616859+0000","last_active":"2024-04-01T13:30:51.638760+0000","last_peered":"2024-04-01T13:30:51.638760+0000","last_clean":"2024-04-01T13:30:51.638760+0000","last_became_active":"2024-04-01T13:30:45.692470+0000","last_became_peered":"2024-04-01T13:30:45.692470+0000","last_unstale":"2024-04-01T13:30:51.638760+0000","last_undegraded":"2024-04-01T13:30:51.638760+0000","last_fullsized":"2024-04-01T13:30:51.638760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":10722734,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":5638,"num_write":26,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"28'26","reported_seq":71,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.696045+0000","last_change":"2024-04-01T13:31:30.697568+0000","last_active":"2024-04-01T13:31:30.696045+0000","last_peered":"2024-04-01T13:31:30.696045+0000","last_clean":"2024-04-01T13:30:29.519276+0000","last_became_active":"2024-04-01T13:31:30.574937+0000","last_became_peered":"2024-04-01T13:31:30.574937+0000","last_unstale":"2024-04-01T13:31:30.696045+0000","last_undegraded":"2024-04-01T13:31:30.696045+0000","last_fullsized":"2024-04-01T13:31:30.696045+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"16'12","reported_seq":86,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.627295+0000","last_change":"2024-04-01T13:31:36.596629+0000","last_active":"2024-04-01T13:31:38.627295+0000","last_peered":"2024-04-01T13:31:38.627295+0000","last_clean":"2024-04-01T13:31:38.627295+0000","last_became_active":"2024-04-01T13:31:30.604705+0000","last_became_peered":"2024-04-01T13:31:30.604705+0000","last_unstale":"2024-04-01T13:31:38.627295+0000","last_undegraded":"2024-04-01T13:31:38.627295+0000","last_fullsized":"2024-04-01T13:31:38.627295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":10843,"num_write":16,"num_write_kb":4752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"24'11","reported_seq":56,"reported_epoch":30,"state":"peering","last_fresh":"2024-04-01T13:31:29.600971+0000","last_change":"2024-04-01T13:31:29.600971+0000","last_active":"2024-04-01T13:31:29.600971+0000","last_peered":"2024-04-01T13:30:49.664233+0000","last_clean":"2024-04-01T13:30:49.664233+0000","last_became_active":"2024-04-01T13:30:30.656628+0000","last_became_peered":"2024-04-01T13:30:30.656628+0000","last_unstale":"2024-04-01T13:31:29.600971+0000","last_undegraded":"2024-04-01T13:31:29.600971+0000","last_fullsized":"2024-04-01T13:31:29.600971+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7495823,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10286,"num_write":13,"num_write_kb":3933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'20","reported_seq":56,"reported_epoch":30,"state":"peering","last_fresh":"2024-04-01T13:31:29.542796+0000","last_change":"2024-04-01T13:31:29.542796+0000","last_active":"2024-04-01T13:31:29.542796+0000","last_peered":"2024-04-01T13:30:49.030107+0000","last_clean":"2024-04-01T13:30:49.030107+0000","last_became_active":"2024-04-01T13:30:30.635917+0000","last_became_peered":"2024-04-01T13:30:30.635917+0000","last_unstale":"2024-04-01T13:31:29.542796+0000","last_undegraded":"2024-04-01T13:31:29.542796+0000","last_fullsized":"2024-04-01T13:31:29.542796+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5865577,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":22,"num_write_kb":6047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"24'20","reported_seq":78,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.697588+0000","last_change":"2024-04-01T13:30:51.647095+0000","last_active":"2024-04-01T13:30:51.697588+0000","last_peered":"2024-04-01T13:30:51.697588+0000","last_clean":"2024-04-01T13:30:51.697588+0000","last_became_active":"2024-04-01T13:30:45.695964+0000","last_became_peered":"2024-04-01T13:30:45.695964+0000","last_unstale":"2024-04-01T13:30:51.697588+0000","last_undegraded":"2024-04-01T13:30:51.697588+0000","last_fullsized":"2024-04-01T13:30:51.697588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11243751,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":14764,"num_write":26,"num_write_kb":7239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":186,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.621853+0000","last_change":"2024-04-01T13:31:30.585158+0000","last_active":"2024-04-01T13:31:38.621853+0000","last_peered":"2024-04-01T13:31:38.621853+0000","last_clean":"2024-04-01T13:31:38.621853+0000","last_became_active":"2024-04-01T13:31:30.578315+0000","last_became_peered":"2024-04-01T13:31:30.578315+0000","last_unstale":"2024-04-01T13:31:38.621853+0000","last_undegraded":"2024-04-01T13:31:38.621853+0000","last_fullsized":"2024-04-01T13:31:38.621853+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_clean_scrub_stamp":"2024-04-01T13:30:22.378617+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.643211+0000","last_change":"2024-04-01T13:30:24.493976+0000","last_active":"2024-04-01T13:30:51.643211+0000","last_peered":"2024-04-01T13:30:51.643211+0000","last_clean":"2024-04-01T13:30:51.643211+0000","last_became_active":"2024-04-01T13:30:24.483528+0000","last_became_peered":"2024-04-01T13:30:24.483528+0000","last_unstale":"2024-04-01T13:30:51.643211+0000","last_undegraded":"2024-04-01T13:30:51.643211+0000","last_fullsized":"2024-04-01T13:30:51.643211+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"24'6","reported_seq":50,"reported_epoch":26,"state":"active+clean+snaptrim","last_fresh":"2024-04-01T13:30:51.640989+0000","last_change":"2024-04-01T13:30:50.619384+0000","last_active":"2024-04-01T13:30:51.640989+0000","last_peered":"2024-04-01T13:30:51.640989+0000","last_clean":"2024-04-01T13:30:51.640989+0000","last_became_active":"2024-04-01T13:30:45.703077+0000","last_became_peered":"2024-04-01T13:30:45.703077+0000","last_unstale":"2024-04-01T13:30:51.640989+0000","last_undegraded":"2024-04-01T13:30:51.640989+0000","last_fullsized":"2024-04-01T13:30:51.640989+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2352947,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"peering","last_fresh":"2024-04-01T13:31:29.731657+0000","last_change":"2024-04-01T13:31:29.731657+0000","last_active":"2024-04-01T13:31:29.731657+0000","last_peered":"2024-04-01T13:30:24.494919+0000","last_clean":"2024-04-01T13:30:24.494919+0000","last_became_active":"2024-04-01T13:30:24.486637+0000","last_became_peered":"2024-04-01T13:30:24.486637+0000","last_unstale":"2024-04-01T13:31:29.731657+0000","last_undegraded":"2024-04-01T13:31:29.731657+0000","last_fullsized":"2024-04-01T13:31:29.731657+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"26'13","reported_seq":67,"reported_epoch":33,"state":"peering","last_fresh":"2024-04-01T13:31:37.629763+0000","last_change":"2024-04-01T13:31:29.552816+0000","last_active":"2024-04-01T13:31:20.179135+0000","last_peered":"2024-04-01T13:30:51.678933+0000","last_clean":"2024-04-01T13:30:51.678933+0000","last_became_active":"2024-04-01T13:30:45.719645+0000","last_became_peered":"2024-04-01T13:30:45.719645+0000","last_unstale":"2024-04-01T13:31:37.629763+0000","last_undegraded":"2024-04-01T13:31:37.629763+0000","last_fullsized":"2024-04-01T13:31:37.629763+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8050454,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10129,"num_write":17,"num_write_kb":4892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":52,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.629011+0000","last_change":"2024-04-01T13:30:45.679818+0000","last_active":"2024-04-01T13:30:51.629011+0000","last_peered":"2024-04-01T13:30:51.629011+0000","last_clean":"2024-04-01T13:30:51.629011+0000","last_became_active":"2024-04-01T13:30:45.674014+0000","last_became_peered":"2024-04-01T13:30:45.674014+0000","last_unstale":"2024-04-01T13:30:51.629011+0000","last_undegraded":"2024-04-01T13:30:51.629011+0000","last_fullsized":"2024-04-01T13:30:51.629011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"26'35","reported_seq":100,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.619593+0000","last_change":"2024-04-01T13:30:53.716341+0000","last_active":"2024-04-01T13:31:38.619593+0000","last_peered":"2024-04-01T13:31:38.619593+0000","last_clean":"2024-04-01T13:31:38.619593+0000","last_became_active":"2024-04-01T13:30:45.733223+0000","last_became_peered":"2024-04-01T13:30:45.733223+0000","last_unstale":"2024-04-01T13:31:38.619593+0000","last_undegraded":"2024-04-01T13:31:38.619593+0000","last_fullsized":"2024-04-01T13:31:38.619593+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":12168103,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":5339,"num_write":35,"num_write_kb":8781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"peering","last_fresh":"2024-04-01T13:31:29.726707+0000","last_change":"2024-04-01T13:31:29.726707+0000","last_active":"2024-04-01T13:31:29.726707+0000","last_peered":"2024-04-01T13:30:24.480593+0000","last_clean":"2024-04-01T13:30:24.480593+0000","last_became_active":"2024-04-01T13:30:24.472751+0000","last_became_peered":"2024-04-01T13:30:24.472751+0000","last_unstale":"2024-04-01T13:31:29.726707+0000","last_undegraded":"2024-04-01T13:31:29.726707+0000","last_fullsized":"2024-04-01T13:31:29.726707+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"26'9","reported_seq":76,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.689469+0000","last_change":"2024-04-01T13:31:30.690519+0000","last_active":"2024-04-01T13:31:30.689469+0000","last_peered":"2024-04-01T13:31:30.689469+0000","last_clean":"2024-04-01T13:30:51.628188+0000","last_became_active":"2024-04-01T13:31:30.573263+0000","last_became_peered":"2024-04-01T13:31:30.573263+0000","last_unstale":"2024-04-01T13:31:30.689469+0000","last_undegraded":"2024-04-01T13:31:30.689469+0000","last_fullsized":"2024-04-01T13:31:30.689469+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"26'15","reported_seq":61,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.656955+0000","last_change":"2024-04-01T13:30:51.714578+0000","last_active":"2024-04-01T13:30:51.656955+0000","last_peered":"2024-04-01T13:30:51.656955+0000","last_clean":"2024-04-01T13:30:51.656955+0000","last_became_active":"2024-04-01T13:30:45.690525+0000","last_became_peered":"2024-04-01T13:30:45.690525+0000","last_unstale":"2024-04-01T13:30:51.656955+0000","last_undegraded":"2024-04-01T13:30:51.656955+0000","last_fullsized":"2024-04-01T13:30:51.656955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067228,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":13,"num_write_kb":2827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.583299+0000","last_change":"2024-04-01T13:31:30.584389+0000","last_active":"2024-04-01T13:31:30.583299+0000","last_peered":"2024-04-01T13:31:30.583299+0000","last_clean":"2024-04-01T13:30:51.632893+0000","last_became_active":"2024-04-01T13:31:30.579930+0000","last_became_peered":"2024-04-01T13:31:30.579930+0000","last_unstale":"2024-04-01T13:31:30.583299+0000","last_undegraded":"2024-04-01T13:31:30.583299+0000","last_fullsized":"2024-04-01T13:31:30.583299+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"16'6","reported_seq":69,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.590018+0000","last_change":"2024-04-01T13:31:30.591443+0000","last_active":"2024-04-01T13:31:30.590018+0000","last_peered":"2024-04-01T13:31:30.590018+0000","last_clean":"2024-04-01T13:30:51.638376+0000","last_became_active":"2024-04-01T13:31:30.581362+0000","last_became_peered":"2024-04-01T13:31:30.581362+0000","last_unstale":"2024-04-01T13:31:30.590018+0000","last_undegraded":"2024-04-01T13:31:30.590018+0000","last_fullsized":"2024-04-01T13:31:30.590018+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4907578,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5282,"num_write":8,"num_write_kb":2191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":56,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.627006+0000","last_change":"2024-04-01T13:31:30.608391+0000","last_active":"2024-04-01T13:31:38.627006+0000","last_peered":"2024-04-01T13:31:38.627006+0000","last_clean":"2024-04-01T13:31:38.627006+0000","last_became_active":"2024-04-01T13:31:30.601821+0000","last_became_peered":"2024-04-01T13:31:30.601821+0000","last_unstale":"2024-04-01T13:31:38.627006+0000","last_undegraded":"2024-04-01T13:31:38.627006+0000","last_fullsized":"2024-04-01T13:31:38.627006+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"24'24","reported_seq":75,"reported_epoch":33,"state":"peering+snaptrim","last_fresh":"2024-04-01T13:31:37.623842+0000","last_change":"2024-04-01T13:31:29.513840+0000","last_active":"2024-04-01T13:31:20.116167+0000","last_peered":"2024-04-01T13:30:51.627795+0000","last_clean":"2024-04-01T13:30:51.627795+0000","last_became_active":"2024-04-01T13:30:45.693544+0000","last_became_peered":"2024-04-01T13:30:45.693544+0000","last_unstale":"2024-04-01T13:31:37.623842+0000","last_undegraded":"2024-04-01T13:31:37.623842+0000","last_fullsized":"2024-04-01T13:31:37.623842+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739541,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":6072,"num_write":29,"num_write_kb":8248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.593772+0000","last_change":"2024-04-01T13:31:30.594953+0000","last_active":"2024-04-01T13:31:30.593772+0000","last_peered":"2024-04-01T13:31:30.593772+0000","last_clean":"2024-04-01T13:30:51.631427+0000","last_became_active":"2024-04-01T13:31:30.586687+0000","last_became_peered":"2024-04-01T13:31:30.586687+0000","last_unstale":"2024-04-01T13:31:30.593772+0000","last_undegraded":"2024-04-01T13:31:30.593772+0000","last_fullsized":"2024-04-01T13:31:30.593772+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-01T13:31:30.532772+0000","last_change":"2024-04-01T13:30:51.650292+0000","last_active":"2024-04-01T13:31:30.532772+0000","last_peered":"2024-04-01T13:31:30.532772+0000","last_clean":"2024-04-01T13:31:30.532772+0000","last_became_active":"2024-04-01T13:30:45.693590+0000","last_became_peered":"2024-04-01T13:30:45.693590+0000","last_unstale":"2024-04-01T13:31:30.532772+0000","last_undegraded":"2024-04-01T13:31:30.532772+0000","last_fullsized":"2024-04-01T13:31:30.532772+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.643299+0000","last_change":"2024-04-01T13:30:24.495962+0000","last_active":"2024-04-01T13:30:51.643299+0000","last_peered":"2024-04-01T13:30:51.643299+0000","last_clean":"2024-04-01T13:30:51.643299+0000","last_became_active":"2024-04-01T13:30:24.489422+0000","last_became_peered":"2024-04-01T13:30:24.489422+0000","last_unstale":"2024-04-01T13:30:51.643299+0000","last_undegraded":"2024-04-01T13:30:51.643299+0000","last_fullsized":"2024-04-01T13:30:51.643299+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":98571801,"num_objects":42,"num_object_clones":4,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":183,"num_read_kb":73219,"num_write":239,"num_write_kb":66094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76812,"kb_used_data":49796,"kb_used_omap":0,"kb_used_meta":27008,"kb_avail":94295028,"statfs":{"total":96636764160,"available":96558108672,"internally_reserved":0,"allocated":50991104,"data_stored":51408199,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27656192},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64556,"kb_used_data":37732,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94307284,"statfs":{"total":96636764160,"available":96570658816,"internally_reserved":0,"allocated":38637568,"data_stored":39479250,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69828,"kb_used_data":42448,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94302012,"statfs":{"total":96636764160,"available":96565260288,"internally_reserved":0,"allocated":43466752,"data_stored":44221816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":29,"seq":124554051589,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73604,"kb_used_data":44696,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94298236,"statfs":{"total":96636764160,"available":96561393664,"internally_reserved":0,"allocated":45768704,"data_stored":47916284,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1508,"internal_metadata":27589148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-01T13:31:42.262 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~16s 2024-04-01T13:31:43.957 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T13:31:44.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:44.094+0000 7f32bc79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:44.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:44.103+0000 7f32bc79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:44.284 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:31:44.285 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-01T13:31:44.296 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-04-01T13:31:43.383956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99162169,"num_objects":44,"num_object_clones":4,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":286,"num_read_kb":73310,"num_write":376,"num_write_kb":67988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":337,"ondisk_log_size":337,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":282892,"kb_used_data":174720,"kb_used_omap":1,"kb_used_meta":107582,"kb_avail":377204468,"statfs":{"total":386547056640,"available":386257375232,"internally_reserved":0,"allocated":178913280,"data_stored":183045187,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":1508,"internal_metadata":110164508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001706"},"pg_stats":[{"pgid":"3.f","version":"16'9","reported_seq":47,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.610970+0000","last_change":"2024-04-01T13:31:29.607744+0000","last_active":"2024-04-01T13:31:29.607744+0000","last_peered":"2024-04-01T13:30:45.695648+0000","last_clean":"2024-04-01T13:30:45.695648+0000","last_became_active":"2024-04-01T13:30:45.686209+0000","last_became_peered":"2024-04-01T13:30:45.686209+0000","last_unstale":"2024-04-01T13:31:38.610970+0000","last_undegraded":"2024-04-01T13:31:38.610970+0000","last_fullsized":"2024-04-01T13:31:38.610970+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7008420,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":4853,"num_write":12,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"26'11","reported_seq":76,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.599421+0000","last_change":"2024-04-01T13:30:53.709220+0000","last_active":"2024-04-01T13:31:39.599421+0000","last_peered":"2024-04-01T13:31:39.599421+0000","last_clean":"2024-04-01T13:31:39.599421+0000","last_became_active":"2024-04-01T13:30:45.689067+0000","last_became_peered":"2024-04-01T13:30:45.689067+0000","last_unstale":"2024-04-01T13:31:39.599421+0000","last_undegraded":"2024-04-01T13:31:39.599421+0000","last_fullsized":"2024-04-01T13:31:39.599421+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042034,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"25'23","reported_seq":81,"reported_epoch":26,"state":"active+clean+snaptrim","last_fresh":"2024-04-01T13:30:51.638760+0000","last_change":"2024-04-01T13:30:50.616859+0000","last_active":"2024-04-01T13:30:51.638760+0000","last_peered":"2024-04-01T13:30:51.638760+0000","last_clean":"2024-04-01T13:30:51.638760+0000","last_became_active":"2024-04-01T13:30:45.692470+0000","last_became_peered":"2024-04-01T13:30:45.692470+0000","last_unstale":"2024-04-01T13:30:51.638760+0000","last_undegraded":"2024-04-01T13:30:51.638760+0000","last_fullsized":"2024-04-01T13:30:51.638760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":10722734,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":5638,"num_write":26,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"28'26","reported_seq":86,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.618774+0000","last_change":"2024-04-01T13:31:36.614767+0000","last_active":"2024-04-01T13:31:39.618774+0000","last_peered":"2024-04-01T13:31:39.618774+0000","last_clean":"2024-04-01T13:31:39.618774+0000","last_became_active":"2024-04-01T13:31:30.574937+0000","last_became_peered":"2024-04-01T13:31:30.574937+0000","last_unstale":"2024-04-01T13:31:39.618774+0000","last_undegraded":"2024-04-01T13:31:39.618774+0000","last_fullsized":"2024-04-01T13:31:39.618774+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"16'12","reported_seq":86,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.627295+0000","last_change":"2024-04-01T13:31:36.596629+0000","last_active":"2024-04-01T13:31:38.627295+0000","last_peered":"2024-04-01T13:31:38.627295+0000","last_clean":"2024-04-01T13:31:38.627295+0000","last_became_active":"2024-04-01T13:31:30.604705+0000","last_became_peered":"2024-04-01T13:31:30.604705+0000","last_unstale":"2024-04-01T13:31:38.627295+0000","last_undegraded":"2024-04-01T13:31:38.627295+0000","last_fullsized":"2024-04-01T13:31:38.627295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":10843,"num_write":16,"num_write_kb":4752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"24'11","reported_seq":60,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.624072+0000","last_change":"2024-04-01T13:31:29.600971+0000","last_active":"2024-04-01T13:31:29.600971+0000","last_peered":"2024-04-01T13:30:49.664233+0000","last_clean":"2024-04-01T13:30:49.664233+0000","last_became_active":"2024-04-01T13:30:30.656628+0000","last_became_peered":"2024-04-01T13:30:30.656628+0000","last_unstale":"2024-04-01T13:31:38.624072+0000","last_undegraded":"2024-04-01T13:31:38.624072+0000","last_fullsized":"2024-04-01T13:31:38.624072+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7495823,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10286,"num_write":13,"num_write_kb":3933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'20","reported_seq":60,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.617062+0000","last_change":"2024-04-01T13:31:29.542796+0000","last_active":"2024-04-01T13:31:29.542796+0000","last_peered":"2024-04-01T13:30:49.030107+0000","last_clean":"2024-04-01T13:30:49.030107+0000","last_became_active":"2024-04-01T13:30:30.635917+0000","last_became_peered":"2024-04-01T13:30:30.635917+0000","last_unstale":"2024-04-01T13:31:38.617062+0000","last_undegraded":"2024-04-01T13:31:38.617062+0000","last_fullsized":"2024-04-01T13:31:38.617062+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5865577,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":22,"num_write_kb":6047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"24'20","reported_seq":78,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.697588+0000","last_change":"2024-04-01T13:30:51.647095+0000","last_active":"2024-04-01T13:30:51.697588+0000","last_peered":"2024-04-01T13:30:51.697588+0000","last_clean":"2024-04-01T13:30:51.697588+0000","last_became_active":"2024-04-01T13:30:45.695964+0000","last_became_peered":"2024-04-01T13:30:45.695964+0000","last_unstale":"2024-04-01T13:30:51.697588+0000","last_undegraded":"2024-04-01T13:30:51.697588+0000","last_fullsized":"2024-04-01T13:30:51.697588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11243751,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":14764,"num_write":26,"num_write_kb":7239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":186,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.621853+0000","last_change":"2024-04-01T13:31:30.585158+0000","last_active":"2024-04-01T13:31:38.621853+0000","last_peered":"2024-04-01T13:31:38.621853+0000","last_clean":"2024-04-01T13:31:38.621853+0000","last_became_active":"2024-04-01T13:31:30.578315+0000","last_became_peered":"2024-04-01T13:31:30.578315+0000","last_unstale":"2024-04-01T13:31:38.621853+0000","last_undegraded":"2024-04-01T13:31:38.621853+0000","last_fullsized":"2024-04-01T13:31:38.621853+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_clean_scrub_stamp":"2024-04-01T13:30:22.378617+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.643211+0000","last_change":"2024-04-01T13:30:24.493976+0000","last_active":"2024-04-01T13:30:51.643211+0000","last_peered":"2024-04-01T13:30:51.643211+0000","last_clean":"2024-04-01T13:30:51.643211+0000","last_became_active":"2024-04-01T13:30:24.483528+0000","last_became_peered":"2024-04-01T13:30:24.483528+0000","last_unstale":"2024-04-01T13:30:51.643211+0000","last_undegraded":"2024-04-01T13:30:51.643211+0000","last_fullsized":"2024-04-01T13:30:51.643211+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"24'6","reported_seq":50,"reported_epoch":26,"state":"active+clean+snaptrim","last_fresh":"2024-04-01T13:30:51.640989+0000","last_change":"2024-04-01T13:30:50.619384+0000","last_active":"2024-04-01T13:30:51.640989+0000","last_peered":"2024-04-01T13:30:51.640989+0000","last_clean":"2024-04-01T13:30:51.640989+0000","last_became_active":"2024-04-01T13:30:45.703077+0000","last_became_peered":"2024-04-01T13:30:45.703077+0000","last_unstale":"2024-04-01T13:30:51.640989+0000","last_undegraded":"2024-04-01T13:30:51.640989+0000","last_fullsized":"2024-04-01T13:30:51.640989+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2352947,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.635036+0000","last_change":"2024-04-01T13:31:29.731657+0000","last_active":"2024-04-01T13:31:29.731657+0000","last_peered":"2024-04-01T13:30:24.494919+0000","last_clean":"2024-04-01T13:30:24.494919+0000","last_became_active":"2024-04-01T13:30:24.486637+0000","last_became_peered":"2024-04-01T13:30:24.486637+0000","last_unstale":"2024-04-01T13:31:38.635036+0000","last_undegraded":"2024-04-01T13:31:38.635036+0000","last_fullsized":"2024-04-01T13:31:38.635036+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"26'13","reported_seq":67,"reported_epoch":33,"state":"peering","last_fresh":"2024-04-01T13:31:37.629763+0000","last_change":"2024-04-01T13:31:29.552816+0000","last_active":"2024-04-01T13:31:20.179135+0000","last_peered":"2024-04-01T13:30:51.678933+0000","last_clean":"2024-04-01T13:30:51.678933+0000","last_became_active":"2024-04-01T13:30:45.719645+0000","last_became_peered":"2024-04-01T13:30:45.719645+0000","last_unstale":"2024-04-01T13:31:37.629763+0000","last_undegraded":"2024-04-01T13:31:37.629763+0000","last_fullsized":"2024-04-01T13:31:37.629763+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8050454,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10129,"num_write":17,"num_write_kb":4892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":52,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.629011+0000","last_change":"2024-04-01T13:30:45.679818+0000","last_active":"2024-04-01T13:30:51.629011+0000","last_peered":"2024-04-01T13:30:51.629011+0000","last_clean":"2024-04-01T13:30:51.629011+0000","last_became_active":"2024-04-01T13:30:45.674014+0000","last_became_peered":"2024-04-01T13:30:45.674014+0000","last_unstale":"2024-04-01T13:30:51.629011+0000","last_undegraded":"2024-04-01T13:30:51.629011+0000","last_fullsized":"2024-04-01T13:30:51.629011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"26'35","reported_seq":100,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.619593+0000","last_change":"2024-04-01T13:30:53.716341+0000","last_active":"2024-04-01T13:31:38.619593+0000","last_peered":"2024-04-01T13:31:38.619593+0000","last_clean":"2024-04-01T13:31:38.619593+0000","last_became_active":"2024-04-01T13:30:45.733223+0000","last_became_peered":"2024-04-01T13:30:45.733223+0000","last_unstale":"2024-04-01T13:31:38.619593+0000","last_undegraded":"2024-04-01T13:31:38.619593+0000","last_fullsized":"2024-04-01T13:31:38.619593+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":12168103,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":5339,"num_write":35,"num_write_kb":8781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.632357+0000","last_change":"2024-04-01T13:31:29.726707+0000","last_active":"2024-04-01T13:31:29.726707+0000","last_peered":"2024-04-01T13:30:24.480593+0000","last_clean":"2024-04-01T13:30:24.480593+0000","last_became_active":"2024-04-01T13:30:24.472751+0000","last_became_peered":"2024-04-01T13:30:24.472751+0000","last_unstale":"2024-04-01T13:31:38.632357+0000","last_undegraded":"2024-04-01T13:31:38.632357+0000","last_fullsized":"2024-04-01T13:31:38.632357+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"26'9","reported_seq":91,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.586336+0000","last_change":"2024-04-01T13:31:37.084731+0000","last_active":"2024-04-01T13:31:39.586336+0000","last_peered":"2024-04-01T13:31:39.586336+0000","last_clean":"2024-04-01T13:31:39.586336+0000","last_became_active":"2024-04-01T13:31:30.573263+0000","last_became_peered":"2024-04-01T13:31:30.573263+0000","last_unstale":"2024-04-01T13:31:39.586336+0000","last_undegraded":"2024-04-01T13:31:39.586336+0000","last_fullsized":"2024-04-01T13:31:39.586336+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"26'15","reported_seq":61,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.656955+0000","last_change":"2024-04-01T13:30:51.714578+0000","last_active":"2024-04-01T13:30:51.656955+0000","last_peered":"2024-04-01T13:30:51.656955+0000","last_clean":"2024-04-01T13:30:51.656955+0000","last_became_active":"2024-04-01T13:30:45.690525+0000","last_became_peered":"2024-04-01T13:30:45.690525+0000","last_unstale":"2024-04-01T13:30:51.656955+0000","last_undegraded":"2024-04-01T13:30:51.656955+0000","last_fullsized":"2024-04-01T13:30:51.656955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067228,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":13,"num_write_kb":2827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":74,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.593086+0000","last_change":"2024-04-01T13:31:30.584389+0000","last_active":"2024-04-01T13:31:39.593086+0000","last_peered":"2024-04-01T13:31:39.593086+0000","last_clean":"2024-04-01T13:31:39.593086+0000","last_became_active":"2024-04-01T13:31:30.579930+0000","last_became_peered":"2024-04-01T13:31:30.579930+0000","last_unstale":"2024-04-01T13:31:39.593086+0000","last_undegraded":"2024-04-01T13:31:39.593086+0000","last_fullsized":"2024-04-01T13:31:39.593086+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"16'6","reported_seq":84,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.594742+0000","last_change":"2024-04-01T13:31:37.090769+0000","last_active":"2024-04-01T13:31:39.594742+0000","last_peered":"2024-04-01T13:31:39.594742+0000","last_clean":"2024-04-01T13:31:39.594742+0000","last_became_active":"2024-04-01T13:31:30.581362+0000","last_became_peered":"2024-04-01T13:31:30.581362+0000","last_unstale":"2024-04-01T13:31:39.594742+0000","last_undegraded":"2024-04-01T13:31:39.594742+0000","last_fullsized":"2024-04-01T13:31:39.594742+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4907578,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5282,"num_write":8,"num_write_kb":2191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":56,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.627006+0000","last_change":"2024-04-01T13:31:30.608391+0000","last_active":"2024-04-01T13:31:38.627006+0000","last_peered":"2024-04-01T13:31:38.627006+0000","last_clean":"2024-04-01T13:31:38.627006+0000","last_became_active":"2024-04-01T13:31:30.601821+0000","last_became_peered":"2024-04-01T13:31:30.601821+0000","last_unstale":"2024-04-01T13:31:38.627006+0000","last_undegraded":"2024-04-01T13:31:38.627006+0000","last_fullsized":"2024-04-01T13:31:38.627006+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"24'24","reported_seq":75,"reported_epoch":33,"state":"peering+snaptrim","last_fresh":"2024-04-01T13:31:37.623842+0000","last_change":"2024-04-01T13:31:29.513840+0000","last_active":"2024-04-01T13:31:20.116167+0000","last_peered":"2024-04-01T13:30:51.627795+0000","last_clean":"2024-04-01T13:30:51.627795+0000","last_became_active":"2024-04-01T13:30:45.693544+0000","last_became_peered":"2024-04-01T13:30:45.693544+0000","last_unstale":"2024-04-01T13:31:37.623842+0000","last_undegraded":"2024-04-01T13:31:37.623842+0000","last_fullsized":"2024-04-01T13:31:37.623842+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739541,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":6072,"num_write":29,"num_write_kb":8248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":74,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.590528+0000","last_change":"2024-04-01T13:31:30.594953+0000","last_active":"2024-04-01T13:31:39.590528+0000","last_peered":"2024-04-01T13:31:39.590528+0000","last_clean":"2024-04-01T13:31:39.590528+0000","last_became_active":"2024-04-01T13:31:30.586687+0000","last_became_peered":"2024-04-01T13:31:30.586687+0000","last_unstale":"2024-04-01T13:31:39.590528+0000","last_undegraded":"2024-04-01T13:31:39.590528+0000","last_fullsized":"2024-04-01T13:31:39.590528+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":66,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.597663+0000","last_change":"2024-04-01T13:30:51.650292+0000","last_active":"2024-04-01T13:31:39.597663+0000","last_peered":"2024-04-01T13:31:39.597663+0000","last_clean":"2024-04-01T13:31:39.597663+0000","last_became_active":"2024-04-01T13:30:45.693590+0000","last_became_peered":"2024-04-01T13:30:45.693590+0000","last_unstale":"2024-04-01T13:31:39.597663+0000","last_undegraded":"2024-04-01T13:31:39.597663+0000","last_fullsized":"2024-04-01T13:31:39.597663+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.643299+0000","last_change":"2024-04-01T13:30:24.495962+0000","last_active":"2024-04-01T13:30:51.643299+0000","last_peered":"2024-04-01T13:30:51.643299+0000","last_clean":"2024-04-01T13:30:51.643299+0000","last_became_active":"2024-04-01T13:30:24.489422+0000","last_became_peered":"2024-04-01T13:30:24.489422+0000","last_unstale":"2024-04-01T13:30:51.643299+0000","last_undegraded":"2024-04-01T13:30:51.643299+0000","last_fullsized":"2024-04-01T13:30:51.643299+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":98571801,"num_objects":42,"num_object_clones":4,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":183,"num_read_kb":73219,"num_write":239,"num_write_kb":66094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76812,"kb_used_data":49796,"kb_used_omap":0,"kb_used_meta":27008,"kb_avail":94295028,"statfs":{"total":96636764160,"available":96558108672,"internally_reserved":0,"allocated":50991104,"data_stored":51408199,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27656192},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64580,"kb_used_data":37756,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94307260,"statfs":{"total":96636764160,"available":96570634240,"internally_reserved":0,"allocated":38662144,"data_stored":39489069,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69828,"kb_used_data":42448,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94302012,"statfs":{"total":96636764160,"available":96565260288,"internally_reserved":0,"allocated":43466752,"data_stored":44221816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":29,"seq":124554051594,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71672,"kb_used_data":44720,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94300168,"statfs":{"total":96636764160,"available":96563372032,"internally_reserved":0,"allocated":45793280,"data_stored":47926103,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1508,"internal_metadata":27589148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-01T13:31:44.297 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-01T13:31:44.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:44.418+0000 7fd70f86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:44.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:44.425+0000 7fd70f86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:44.680 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:31:44.680 INFO:teuthology.orchestra.run.smithi059.stdout:{"fsid":"2a626e12-75d1-4fa9-b59b-fc64723c30fe","health":{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 2 pgs peering","count":2},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":91,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1711978289,"num_in_osds":4,"osd_in_since":1711978215,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"peering","count":6},{"state_name":"active+clean+snaptrim","count":2},{"state_name":"peering+snaptrim","count":1}],"num_pgs":25,"num_pools":3,"num_objects":44,"data_bytes":99162169,"bytes_used":289681408,"bytes_avail":386257375232,"bytes_total":386547056640,"inactive_pgs_ratio":0.2800000011920929},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-01T13:31:19.377894+0000","services":{}},"progress_events":{}} 2024-04-01T13:31:44.692 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-01T13:31:44.692 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T13:31:44.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:44.815+0000 7f86ee06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:44.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:44.823+0000 7f86ee06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:45.006 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:31:45.006 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-01T13:31:45.018 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-04-01T13:31:43.383956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99162169,"num_objects":44,"num_object_clones":4,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":286,"num_read_kb":73310,"num_write":376,"num_write_kb":67988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":337,"ondisk_log_size":337,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":282892,"kb_used_data":174720,"kb_used_omap":1,"kb_used_meta":107582,"kb_avail":377204468,"statfs":{"total":386547056640,"available":386257375232,"internally_reserved":0,"allocated":178913280,"data_stored":183045187,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":1508,"internal_metadata":110164508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001706"},"pg_stats":[{"pgid":"3.f","version":"16'9","reported_seq":47,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.610970+0000","last_change":"2024-04-01T13:31:29.607744+0000","last_active":"2024-04-01T13:31:29.607744+0000","last_peered":"2024-04-01T13:30:45.695648+0000","last_clean":"2024-04-01T13:30:45.695648+0000","last_became_active":"2024-04-01T13:30:45.686209+0000","last_became_peered":"2024-04-01T13:30:45.686209+0000","last_unstale":"2024-04-01T13:31:38.610970+0000","last_undegraded":"2024-04-01T13:31:38.610970+0000","last_fullsized":"2024-04-01T13:31:38.610970+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7008420,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":4853,"num_write":12,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"26'11","reported_seq":76,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.599421+0000","last_change":"2024-04-01T13:30:53.709220+0000","last_active":"2024-04-01T13:31:39.599421+0000","last_peered":"2024-04-01T13:31:39.599421+0000","last_clean":"2024-04-01T13:31:39.599421+0000","last_became_active":"2024-04-01T13:30:45.689067+0000","last_became_peered":"2024-04-01T13:30:45.689067+0000","last_unstale":"2024-04-01T13:31:39.599421+0000","last_undegraded":"2024-04-01T13:31:39.599421+0000","last_fullsized":"2024-04-01T13:31:39.599421+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042034,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"25'23","reported_seq":81,"reported_epoch":26,"state":"active+clean+snaptrim","last_fresh":"2024-04-01T13:30:51.638760+0000","last_change":"2024-04-01T13:30:50.616859+0000","last_active":"2024-04-01T13:30:51.638760+0000","last_peered":"2024-04-01T13:30:51.638760+0000","last_clean":"2024-04-01T13:30:51.638760+0000","last_became_active":"2024-04-01T13:30:45.692470+0000","last_became_peered":"2024-04-01T13:30:45.692470+0000","last_unstale":"2024-04-01T13:30:51.638760+0000","last_undegraded":"2024-04-01T13:30:51.638760+0000","last_fullsized":"2024-04-01T13:30:51.638760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":10722734,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":5638,"num_write":26,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"28'26","reported_seq":86,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.618774+0000","last_change":"2024-04-01T13:31:36.614767+0000","last_active":"2024-04-01T13:31:39.618774+0000","last_peered":"2024-04-01T13:31:39.618774+0000","last_clean":"2024-04-01T13:31:39.618774+0000","last_became_active":"2024-04-01T13:31:30.574937+0000","last_became_peered":"2024-04-01T13:31:30.574937+0000","last_unstale":"2024-04-01T13:31:39.618774+0000","last_undegraded":"2024-04-01T13:31:39.618774+0000","last_fullsized":"2024-04-01T13:31:39.618774+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"16'12","reported_seq":86,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.627295+0000","last_change":"2024-04-01T13:31:36.596629+0000","last_active":"2024-04-01T13:31:38.627295+0000","last_peered":"2024-04-01T13:31:38.627295+0000","last_clean":"2024-04-01T13:31:38.627295+0000","last_became_active":"2024-04-01T13:31:30.604705+0000","last_became_peered":"2024-04-01T13:31:30.604705+0000","last_unstale":"2024-04-01T13:31:38.627295+0000","last_undegraded":"2024-04-01T13:31:38.627295+0000","last_fullsized":"2024-04-01T13:31:38.627295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":10843,"num_write":16,"num_write_kb":4752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"24'11","reported_seq":60,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.624072+0000","last_change":"2024-04-01T13:31:29.600971+0000","last_active":"2024-04-01T13:31:29.600971+0000","last_peered":"2024-04-01T13:30:49.664233+0000","last_clean":"2024-04-01T13:30:49.664233+0000","last_became_active":"2024-04-01T13:30:30.656628+0000","last_became_peered":"2024-04-01T13:30:30.656628+0000","last_unstale":"2024-04-01T13:31:38.624072+0000","last_undegraded":"2024-04-01T13:31:38.624072+0000","last_fullsized":"2024-04-01T13:31:38.624072+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7495823,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10286,"num_write":13,"num_write_kb":3933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'20","reported_seq":60,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.617062+0000","last_change":"2024-04-01T13:31:29.542796+0000","last_active":"2024-04-01T13:31:29.542796+0000","last_peered":"2024-04-01T13:30:49.030107+0000","last_clean":"2024-04-01T13:30:49.030107+0000","last_became_active":"2024-04-01T13:30:30.635917+0000","last_became_peered":"2024-04-01T13:30:30.635917+0000","last_unstale":"2024-04-01T13:31:38.617062+0000","last_undegraded":"2024-04-01T13:31:38.617062+0000","last_fullsized":"2024-04-01T13:31:38.617062+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5865577,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":22,"num_write_kb":6047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"24'20","reported_seq":78,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.697588+0000","last_change":"2024-04-01T13:30:51.647095+0000","last_active":"2024-04-01T13:30:51.697588+0000","last_peered":"2024-04-01T13:30:51.697588+0000","last_clean":"2024-04-01T13:30:51.697588+0000","last_became_active":"2024-04-01T13:30:45.695964+0000","last_became_peered":"2024-04-01T13:30:45.695964+0000","last_unstale":"2024-04-01T13:30:51.697588+0000","last_undegraded":"2024-04-01T13:30:51.697588+0000","last_fullsized":"2024-04-01T13:30:51.697588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11243751,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":14764,"num_write":26,"num_write_kb":7239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":186,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.621853+0000","last_change":"2024-04-01T13:31:30.585158+0000","last_active":"2024-04-01T13:31:38.621853+0000","last_peered":"2024-04-01T13:31:38.621853+0000","last_clean":"2024-04-01T13:31:38.621853+0000","last_became_active":"2024-04-01T13:31:30.578315+0000","last_became_peered":"2024-04-01T13:31:30.578315+0000","last_unstale":"2024-04-01T13:31:38.621853+0000","last_undegraded":"2024-04-01T13:31:38.621853+0000","last_fullsized":"2024-04-01T13:31:38.621853+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_clean_scrub_stamp":"2024-04-01T13:30:22.378617+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.643211+0000","last_change":"2024-04-01T13:30:24.493976+0000","last_active":"2024-04-01T13:30:51.643211+0000","last_peered":"2024-04-01T13:30:51.643211+0000","last_clean":"2024-04-01T13:30:51.643211+0000","last_became_active":"2024-04-01T13:30:24.483528+0000","last_became_peered":"2024-04-01T13:30:24.483528+0000","last_unstale":"2024-04-01T13:30:51.643211+0000","last_undegraded":"2024-04-01T13:30:51.643211+0000","last_fullsized":"2024-04-01T13:30:51.643211+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"24'6","reported_seq":50,"reported_epoch":26,"state":"active+clean+snaptrim","last_fresh":"2024-04-01T13:30:51.640989+0000","last_change":"2024-04-01T13:30:50.619384+0000","last_active":"2024-04-01T13:30:51.640989+0000","last_peered":"2024-04-01T13:30:51.640989+0000","last_clean":"2024-04-01T13:30:51.640989+0000","last_became_active":"2024-04-01T13:30:45.703077+0000","last_became_peered":"2024-04-01T13:30:45.703077+0000","last_unstale":"2024-04-01T13:30:51.640989+0000","last_undegraded":"2024-04-01T13:30:51.640989+0000","last_fullsized":"2024-04-01T13:30:51.640989+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2352947,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.635036+0000","last_change":"2024-04-01T13:31:29.731657+0000","last_active":"2024-04-01T13:31:29.731657+0000","last_peered":"2024-04-01T13:30:24.494919+0000","last_clean":"2024-04-01T13:30:24.494919+0000","last_became_active":"2024-04-01T13:30:24.486637+0000","last_became_peered":"2024-04-01T13:30:24.486637+0000","last_unstale":"2024-04-01T13:31:38.635036+0000","last_undegraded":"2024-04-01T13:31:38.635036+0000","last_fullsized":"2024-04-01T13:31:38.635036+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"26'13","reported_seq":67,"reported_epoch":33,"state":"peering","last_fresh":"2024-04-01T13:31:37.629763+0000","last_change":"2024-04-01T13:31:29.552816+0000","last_active":"2024-04-01T13:31:20.179135+0000","last_peered":"2024-04-01T13:30:51.678933+0000","last_clean":"2024-04-01T13:30:51.678933+0000","last_became_active":"2024-04-01T13:30:45.719645+0000","last_became_peered":"2024-04-01T13:30:45.719645+0000","last_unstale":"2024-04-01T13:31:37.629763+0000","last_undegraded":"2024-04-01T13:31:37.629763+0000","last_fullsized":"2024-04-01T13:31:37.629763+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8050454,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10129,"num_write":17,"num_write_kb":4892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":52,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.629011+0000","last_change":"2024-04-01T13:30:45.679818+0000","last_active":"2024-04-01T13:30:51.629011+0000","last_peered":"2024-04-01T13:30:51.629011+0000","last_clean":"2024-04-01T13:30:51.629011+0000","last_became_active":"2024-04-01T13:30:45.674014+0000","last_became_peered":"2024-04-01T13:30:45.674014+0000","last_unstale":"2024-04-01T13:30:51.629011+0000","last_undegraded":"2024-04-01T13:30:51.629011+0000","last_fullsized":"2024-04-01T13:30:51.629011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"26'35","reported_seq":100,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.619593+0000","last_change":"2024-04-01T13:30:53.716341+0000","last_active":"2024-04-01T13:31:38.619593+0000","last_peered":"2024-04-01T13:31:38.619593+0000","last_clean":"2024-04-01T13:31:38.619593+0000","last_became_active":"2024-04-01T13:30:45.733223+0000","last_became_peered":"2024-04-01T13:30:45.733223+0000","last_unstale":"2024-04-01T13:31:38.619593+0000","last_undegraded":"2024-04-01T13:31:38.619593+0000","last_fullsized":"2024-04-01T13:31:38.619593+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":12168103,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":5339,"num_write":35,"num_write_kb":8781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.632357+0000","last_change":"2024-04-01T13:31:29.726707+0000","last_active":"2024-04-01T13:31:29.726707+0000","last_peered":"2024-04-01T13:30:24.480593+0000","last_clean":"2024-04-01T13:30:24.480593+0000","last_became_active":"2024-04-01T13:30:24.472751+0000","last_became_peered":"2024-04-01T13:30:24.472751+0000","last_unstale":"2024-04-01T13:31:38.632357+0000","last_undegraded":"2024-04-01T13:31:38.632357+0000","last_fullsized":"2024-04-01T13:31:38.632357+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"26'9","reported_seq":91,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.586336+0000","last_change":"2024-04-01T13:31:37.084731+0000","last_active":"2024-04-01T13:31:39.586336+0000","last_peered":"2024-04-01T13:31:39.586336+0000","last_clean":"2024-04-01T13:31:39.586336+0000","last_became_active":"2024-04-01T13:31:30.573263+0000","last_became_peered":"2024-04-01T13:31:30.573263+0000","last_unstale":"2024-04-01T13:31:39.586336+0000","last_undegraded":"2024-04-01T13:31:39.586336+0000","last_fullsized":"2024-04-01T13:31:39.586336+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"26'15","reported_seq":61,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.656955+0000","last_change":"2024-04-01T13:30:51.714578+0000","last_active":"2024-04-01T13:30:51.656955+0000","last_peered":"2024-04-01T13:30:51.656955+0000","last_clean":"2024-04-01T13:30:51.656955+0000","last_became_active":"2024-04-01T13:30:45.690525+0000","last_became_peered":"2024-04-01T13:30:45.690525+0000","last_unstale":"2024-04-01T13:30:51.656955+0000","last_undegraded":"2024-04-01T13:30:51.656955+0000","last_fullsized":"2024-04-01T13:30:51.656955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067228,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":13,"num_write_kb":2827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":74,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.593086+0000","last_change":"2024-04-01T13:31:30.584389+0000","last_active":"2024-04-01T13:31:39.593086+0000","last_peered":"2024-04-01T13:31:39.593086+0000","last_clean":"2024-04-01T13:31:39.593086+0000","last_became_active":"2024-04-01T13:31:30.579930+0000","last_became_peered":"2024-04-01T13:31:30.579930+0000","last_unstale":"2024-04-01T13:31:39.593086+0000","last_undegraded":"2024-04-01T13:31:39.593086+0000","last_fullsized":"2024-04-01T13:31:39.593086+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"16'6","reported_seq":84,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.594742+0000","last_change":"2024-04-01T13:31:37.090769+0000","last_active":"2024-04-01T13:31:39.594742+0000","last_peered":"2024-04-01T13:31:39.594742+0000","last_clean":"2024-04-01T13:31:39.594742+0000","last_became_active":"2024-04-01T13:31:30.581362+0000","last_became_peered":"2024-04-01T13:31:30.581362+0000","last_unstale":"2024-04-01T13:31:39.594742+0000","last_undegraded":"2024-04-01T13:31:39.594742+0000","last_fullsized":"2024-04-01T13:31:39.594742+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4907578,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5282,"num_write":8,"num_write_kb":2191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":56,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-01T13:31:38.627006+0000","last_change":"2024-04-01T13:31:30.608391+0000","last_active":"2024-04-01T13:31:38.627006+0000","last_peered":"2024-04-01T13:31:38.627006+0000","last_clean":"2024-04-01T13:31:38.627006+0000","last_became_active":"2024-04-01T13:31:30.601821+0000","last_became_peered":"2024-04-01T13:31:30.601821+0000","last_unstale":"2024-04-01T13:31:38.627006+0000","last_undegraded":"2024-04-01T13:31:38.627006+0000","last_fullsized":"2024-04-01T13:31:38.627006+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"24'24","reported_seq":75,"reported_epoch":33,"state":"peering+snaptrim","last_fresh":"2024-04-01T13:31:37.623842+0000","last_change":"2024-04-01T13:31:29.513840+0000","last_active":"2024-04-01T13:31:20.116167+0000","last_peered":"2024-04-01T13:30:51.627795+0000","last_clean":"2024-04-01T13:30:51.627795+0000","last_became_active":"2024-04-01T13:30:45.693544+0000","last_became_peered":"2024-04-01T13:30:45.693544+0000","last_unstale":"2024-04-01T13:31:37.623842+0000","last_undegraded":"2024-04-01T13:31:37.623842+0000","last_fullsized":"2024-04-01T13:31:37.623842+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739541,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":6072,"num_write":29,"num_write_kb":8248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":74,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.590528+0000","last_change":"2024-04-01T13:31:30.594953+0000","last_active":"2024-04-01T13:31:39.590528+0000","last_peered":"2024-04-01T13:31:39.590528+0000","last_clean":"2024-04-01T13:31:39.590528+0000","last_became_active":"2024-04-01T13:31:30.586687+0000","last_became_peered":"2024-04-01T13:31:30.586687+0000","last_unstale":"2024-04-01T13:31:39.590528+0000","last_undegraded":"2024-04-01T13:31:39.590528+0000","last_fullsized":"2024-04-01T13:31:39.590528+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":66,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.597663+0000","last_change":"2024-04-01T13:30:51.650292+0000","last_active":"2024-04-01T13:31:39.597663+0000","last_peered":"2024-04-01T13:31:39.597663+0000","last_clean":"2024-04-01T13:31:39.597663+0000","last_became_active":"2024-04-01T13:30:45.693590+0000","last_became_peered":"2024-04-01T13:30:45.693590+0000","last_unstale":"2024-04-01T13:31:39.597663+0000","last_undegraded":"2024-04-01T13:31:39.597663+0000","last_fullsized":"2024-04-01T13:31:39.597663+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.643299+0000","last_change":"2024-04-01T13:30:24.495962+0000","last_active":"2024-04-01T13:30:51.643299+0000","last_peered":"2024-04-01T13:30:51.643299+0000","last_clean":"2024-04-01T13:30:51.643299+0000","last_became_active":"2024-04-01T13:30:24.489422+0000","last_became_peered":"2024-04-01T13:30:24.489422+0000","last_unstale":"2024-04-01T13:30:51.643299+0000","last_undegraded":"2024-04-01T13:30:51.643299+0000","last_fullsized":"2024-04-01T13:30:51.643299+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":98571801,"num_objects":42,"num_object_clones":4,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":183,"num_read_kb":73219,"num_write":239,"num_write_kb":66094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76812,"kb_used_data":49796,"kb_used_omap":0,"kb_used_meta":27008,"kb_avail":94295028,"statfs":{"total":96636764160,"available":96558108672,"internally_reserved":0,"allocated":50991104,"data_stored":51408199,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27656192},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64580,"kb_used_data":37756,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94307260,"statfs":{"total":96636764160,"available":96570634240,"internally_reserved":0,"allocated":38662144,"data_stored":39489069,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69828,"kb_used_data":42448,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94302012,"statfs":{"total":96636764160,"available":96565260288,"internally_reserved":0,"allocated":43466752,"data_stored":44221816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":29,"seq":124554051594,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71672,"kb_used_data":44720,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94300168,"statfs":{"total":96636764160,"available":96563372032,"internally_reserved":0,"allocated":45793280,"data_stored":47926103,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1508,"internal_metadata":27589148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-01T13:31:47.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~22s 2024-04-01T13:31:48.020 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T13:31:48.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:48.156+0000 7f4ded110640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:48.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:48.164+0000 7f4ded110640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:48.350 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:31:48.350 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-01T13:31:48.362 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-04-01T13:31:47.385078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99162169,"num_objects":44,"num_object_clones":4,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":286,"num_read_kb":73310,"num_write":376,"num_write_kb":67988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":337,"ondisk_log_size":337,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":282924,"kb_used_data":174752,"kb_used_omap":1,"kb_used_meta":107582,"kb_avail":377204436,"statfs":{"total":386547056640,"available":386257342464,"internally_reserved":0,"allocated":178946048,"data_stored":183058279,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":1508,"internal_metadata":110164508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.820633"},"pg_stats":[{"pgid":"3.f","version":"16'9","reported_seq":47,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.610970+0000","last_change":"2024-04-01T13:31:29.607744+0000","last_active":"2024-04-01T13:31:29.607744+0000","last_peered":"2024-04-01T13:30:45.695648+0000","last_clean":"2024-04-01T13:30:45.695648+0000","last_became_active":"2024-04-01T13:30:45.686209+0000","last_became_peered":"2024-04-01T13:30:45.686209+0000","last_unstale":"2024-04-01T13:31:38.610970+0000","last_undegraded":"2024-04-01T13:31:38.610970+0000","last_fullsized":"2024-04-01T13:31:38.610970+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7008420,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":4853,"num_write":12,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"26'11","reported_seq":76,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.599421+0000","last_change":"2024-04-01T13:30:53.709220+0000","last_active":"2024-04-01T13:31:39.599421+0000","last_peered":"2024-04-01T13:31:39.599421+0000","last_clean":"2024-04-01T13:31:39.599421+0000","last_became_active":"2024-04-01T13:30:45.689067+0000","last_became_peered":"2024-04-01T13:30:45.689067+0000","last_unstale":"2024-04-01T13:31:39.599421+0000","last_undegraded":"2024-04-01T13:31:39.599421+0000","last_fullsized":"2024-04-01T13:31:39.599421+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042034,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"25'23","reported_seq":81,"reported_epoch":26,"state":"active+clean+snaptrim","last_fresh":"2024-04-01T13:30:51.638760+0000","last_change":"2024-04-01T13:30:50.616859+0000","last_active":"2024-04-01T13:30:51.638760+0000","last_peered":"2024-04-01T13:30:51.638760+0000","last_clean":"2024-04-01T13:30:51.638760+0000","last_became_active":"2024-04-01T13:30:45.692470+0000","last_became_peered":"2024-04-01T13:30:45.692470+0000","last_unstale":"2024-04-01T13:30:51.638760+0000","last_undegraded":"2024-04-01T13:30:51.638760+0000","last_fullsized":"2024-04-01T13:30:51.638760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":10722734,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":5638,"num_write":26,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"28'26","reported_seq":86,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.618774+0000","last_change":"2024-04-01T13:31:36.614767+0000","last_active":"2024-04-01T13:31:39.618774+0000","last_peered":"2024-04-01T13:31:39.618774+0000","last_clean":"2024-04-01T13:31:39.618774+0000","last_became_active":"2024-04-01T13:31:30.574937+0000","last_became_peered":"2024-04-01T13:31:30.574937+0000","last_unstale":"2024-04-01T13:31:39.618774+0000","last_undegraded":"2024-04-01T13:31:39.618774+0000","last_fullsized":"2024-04-01T13:31:39.618774+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"16'12","reported_seq":89,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.638600+0000","last_change":"2024-04-01T13:31:36.596629+0000","last_active":"2024-04-01T13:31:39.638600+0000","last_peered":"2024-04-01T13:31:39.638600+0000","last_clean":"2024-04-01T13:31:39.638600+0000","last_became_active":"2024-04-01T13:31:30.604705+0000","last_became_peered":"2024-04-01T13:31:30.604705+0000","last_unstale":"2024-04-01T13:31:39.638600+0000","last_undegraded":"2024-04-01T13:31:39.638600+0000","last_fullsized":"2024-04-01T13:31:39.638600+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":10843,"num_write":16,"num_write_kb":4752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"24'11","reported_seq":60,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.624072+0000","last_change":"2024-04-01T13:31:29.600971+0000","last_active":"2024-04-01T13:31:29.600971+0000","last_peered":"2024-04-01T13:30:49.664233+0000","last_clean":"2024-04-01T13:30:49.664233+0000","last_became_active":"2024-04-01T13:30:30.656628+0000","last_became_peered":"2024-04-01T13:30:30.656628+0000","last_unstale":"2024-04-01T13:31:38.624072+0000","last_undegraded":"2024-04-01T13:31:38.624072+0000","last_fullsized":"2024-04-01T13:31:38.624072+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7495823,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10286,"num_write":13,"num_write_kb":3933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'20","reported_seq":60,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.617062+0000","last_change":"2024-04-01T13:31:29.542796+0000","last_active":"2024-04-01T13:31:29.542796+0000","last_peered":"2024-04-01T13:30:49.030107+0000","last_clean":"2024-04-01T13:30:49.030107+0000","last_became_active":"2024-04-01T13:30:30.635917+0000","last_became_peered":"2024-04-01T13:30:30.635917+0000","last_unstale":"2024-04-01T13:31:38.617062+0000","last_undegraded":"2024-04-01T13:31:38.617062+0000","last_fullsized":"2024-04-01T13:31:38.617062+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5865577,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":22,"num_write_kb":6047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"24'20","reported_seq":78,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.697588+0000","last_change":"2024-04-01T13:30:51.647095+0000","last_active":"2024-04-01T13:30:51.697588+0000","last_peered":"2024-04-01T13:30:51.697588+0000","last_clean":"2024-04-01T13:30:51.697588+0000","last_became_active":"2024-04-01T13:30:45.695964+0000","last_became_peered":"2024-04-01T13:30:45.695964+0000","last_unstale":"2024-04-01T13:30:51.697588+0000","last_undegraded":"2024-04-01T13:30:51.697588+0000","last_fullsized":"2024-04-01T13:30:51.697588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11243751,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":14764,"num_write":26,"num_write_kb":7239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":189,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.627624+0000","last_change":"2024-04-01T13:31:30.585158+0000","last_active":"2024-04-01T13:31:39.627624+0000","last_peered":"2024-04-01T13:31:39.627624+0000","last_clean":"2024-04-01T13:31:39.627624+0000","last_became_active":"2024-04-01T13:31:30.578315+0000","last_became_peered":"2024-04-01T13:31:30.578315+0000","last_unstale":"2024-04-01T13:31:39.627624+0000","last_undegraded":"2024-04-01T13:31:39.627624+0000","last_fullsized":"2024-04-01T13:31:39.627624+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_clean_scrub_stamp":"2024-04-01T13:30:22.378617+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.643211+0000","last_change":"2024-04-01T13:30:24.493976+0000","last_active":"2024-04-01T13:30:51.643211+0000","last_peered":"2024-04-01T13:30:51.643211+0000","last_clean":"2024-04-01T13:30:51.643211+0000","last_became_active":"2024-04-01T13:30:24.483528+0000","last_became_peered":"2024-04-01T13:30:24.483528+0000","last_unstale":"2024-04-01T13:30:51.643211+0000","last_undegraded":"2024-04-01T13:30:51.643211+0000","last_fullsized":"2024-04-01T13:30:51.643211+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"24'6","reported_seq":50,"reported_epoch":26,"state":"active+clean+snaptrim","last_fresh":"2024-04-01T13:30:51.640989+0000","last_change":"2024-04-01T13:30:50.619384+0000","last_active":"2024-04-01T13:30:51.640989+0000","last_peered":"2024-04-01T13:30:51.640989+0000","last_clean":"2024-04-01T13:30:51.640989+0000","last_became_active":"2024-04-01T13:30:45.703077+0000","last_became_peered":"2024-04-01T13:30:45.703077+0000","last_unstale":"2024-04-01T13:30:51.640989+0000","last_undegraded":"2024-04-01T13:30:51.640989+0000","last_fullsized":"2024-04-01T13:30:51.640989+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2352947,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.635036+0000","last_change":"2024-04-01T13:31:29.731657+0000","last_active":"2024-04-01T13:31:29.731657+0000","last_peered":"2024-04-01T13:30:24.494919+0000","last_clean":"2024-04-01T13:30:24.494919+0000","last_became_active":"2024-04-01T13:30:24.486637+0000","last_became_peered":"2024-04-01T13:30:24.486637+0000","last_unstale":"2024-04-01T13:31:38.635036+0000","last_undegraded":"2024-04-01T13:31:38.635036+0000","last_fullsized":"2024-04-01T13:31:38.635036+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"26'13","reported_seq":68,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.631922+0000","last_change":"2024-04-01T13:31:29.552816+0000","last_active":"2024-04-01T13:31:20.179135+0000","last_peered":"2024-04-01T13:30:51.678933+0000","last_clean":"2024-04-01T13:30:51.678933+0000","last_became_active":"2024-04-01T13:30:45.719645+0000","last_became_peered":"2024-04-01T13:30:45.719645+0000","last_unstale":"2024-04-01T13:31:38.631922+0000","last_undegraded":"2024-04-01T13:31:38.631922+0000","last_fullsized":"2024-04-01T13:31:38.631922+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8050454,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10129,"num_write":17,"num_write_kb":4892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":52,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.629011+0000","last_change":"2024-04-01T13:30:45.679818+0000","last_active":"2024-04-01T13:30:51.629011+0000","last_peered":"2024-04-01T13:30:51.629011+0000","last_clean":"2024-04-01T13:30:51.629011+0000","last_became_active":"2024-04-01T13:30:45.674014+0000","last_became_peered":"2024-04-01T13:30:45.674014+0000","last_unstale":"2024-04-01T13:30:51.629011+0000","last_undegraded":"2024-04-01T13:30:51.629011+0000","last_fullsized":"2024-04-01T13:30:51.629011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"26'35","reported_seq":103,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.629586+0000","last_change":"2024-04-01T13:30:53.716341+0000","last_active":"2024-04-01T13:31:39.629586+0000","last_peered":"2024-04-01T13:31:39.629586+0000","last_clean":"2024-04-01T13:31:39.629586+0000","last_became_active":"2024-04-01T13:30:45.733223+0000","last_became_peered":"2024-04-01T13:30:45.733223+0000","last_unstale":"2024-04-01T13:31:39.629586+0000","last_undegraded":"2024-04-01T13:31:39.629586+0000","last_fullsized":"2024-04-01T13:31:39.629586+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":12168103,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":5339,"num_write":35,"num_write_kb":8781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.632357+0000","last_change":"2024-04-01T13:31:29.726707+0000","last_active":"2024-04-01T13:31:29.726707+0000","last_peered":"2024-04-01T13:30:24.480593+0000","last_clean":"2024-04-01T13:30:24.480593+0000","last_became_active":"2024-04-01T13:30:24.472751+0000","last_became_peered":"2024-04-01T13:30:24.472751+0000","last_unstale":"2024-04-01T13:31:38.632357+0000","last_undegraded":"2024-04-01T13:31:38.632357+0000","last_fullsized":"2024-04-01T13:31:38.632357+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"26'9","reported_seq":91,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.586336+0000","last_change":"2024-04-01T13:31:37.084731+0000","last_active":"2024-04-01T13:31:39.586336+0000","last_peered":"2024-04-01T13:31:39.586336+0000","last_clean":"2024-04-01T13:31:39.586336+0000","last_became_active":"2024-04-01T13:31:30.573263+0000","last_became_peered":"2024-04-01T13:31:30.573263+0000","last_unstale":"2024-04-01T13:31:39.586336+0000","last_undegraded":"2024-04-01T13:31:39.586336+0000","last_fullsized":"2024-04-01T13:31:39.586336+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"26'15","reported_seq":61,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.656955+0000","last_change":"2024-04-01T13:30:51.714578+0000","last_active":"2024-04-01T13:30:51.656955+0000","last_peered":"2024-04-01T13:30:51.656955+0000","last_clean":"2024-04-01T13:30:51.656955+0000","last_became_active":"2024-04-01T13:30:45.690525+0000","last_became_peered":"2024-04-01T13:30:45.690525+0000","last_unstale":"2024-04-01T13:30:51.656955+0000","last_undegraded":"2024-04-01T13:30:51.656955+0000","last_fullsized":"2024-04-01T13:30:51.656955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067228,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":13,"num_write_kb":2827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":74,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.593086+0000","last_change":"2024-04-01T13:31:30.584389+0000","last_active":"2024-04-01T13:31:39.593086+0000","last_peered":"2024-04-01T13:31:39.593086+0000","last_clean":"2024-04-01T13:31:39.593086+0000","last_became_active":"2024-04-01T13:31:30.579930+0000","last_became_peered":"2024-04-01T13:31:30.579930+0000","last_unstale":"2024-04-01T13:31:39.593086+0000","last_undegraded":"2024-04-01T13:31:39.593086+0000","last_fullsized":"2024-04-01T13:31:39.593086+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"16'6","reported_seq":84,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.594742+0000","last_change":"2024-04-01T13:31:37.090769+0000","last_active":"2024-04-01T13:31:39.594742+0000","last_peered":"2024-04-01T13:31:39.594742+0000","last_clean":"2024-04-01T13:31:39.594742+0000","last_became_active":"2024-04-01T13:31:30.581362+0000","last_became_peered":"2024-04-01T13:31:30.581362+0000","last_unstale":"2024-04-01T13:31:39.594742+0000","last_undegraded":"2024-04-01T13:31:39.594742+0000","last_fullsized":"2024-04-01T13:31:39.594742+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4907578,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5282,"num_write":8,"num_write_kb":2191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.639590+0000","last_change":"2024-04-01T13:31:30.608391+0000","last_active":"2024-04-01T13:31:39.639590+0000","last_peered":"2024-04-01T13:31:39.639590+0000","last_clean":"2024-04-01T13:31:39.639590+0000","last_became_active":"2024-04-01T13:31:30.601821+0000","last_became_peered":"2024-04-01T13:31:30.601821+0000","last_unstale":"2024-04-01T13:31:39.639590+0000","last_undegraded":"2024-04-01T13:31:39.639590+0000","last_fullsized":"2024-04-01T13:31:39.639590+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"24'24","reported_seq":76,"reported_epoch":34,"state":"peering+snaptrim","last_fresh":"2024-04-01T13:31:38.620596+0000","last_change":"2024-04-01T13:31:29.513840+0000","last_active":"2024-04-01T13:31:20.116167+0000","last_peered":"2024-04-01T13:30:51.627795+0000","last_clean":"2024-04-01T13:30:51.627795+0000","last_became_active":"2024-04-01T13:30:45.693544+0000","last_became_peered":"2024-04-01T13:30:45.693544+0000","last_unstale":"2024-04-01T13:31:38.620596+0000","last_undegraded":"2024-04-01T13:31:38.620596+0000","last_fullsized":"2024-04-01T13:31:38.620596+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739541,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":6072,"num_write":29,"num_write_kb":8248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":74,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.590528+0000","last_change":"2024-04-01T13:31:30.594953+0000","last_active":"2024-04-01T13:31:39.590528+0000","last_peered":"2024-04-01T13:31:39.590528+0000","last_clean":"2024-04-01T13:31:39.590528+0000","last_became_active":"2024-04-01T13:31:30.586687+0000","last_became_peered":"2024-04-01T13:31:30.586687+0000","last_unstale":"2024-04-01T13:31:39.590528+0000","last_undegraded":"2024-04-01T13:31:39.590528+0000","last_fullsized":"2024-04-01T13:31:39.590528+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":66,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.597663+0000","last_change":"2024-04-01T13:30:51.650292+0000","last_active":"2024-04-01T13:31:39.597663+0000","last_peered":"2024-04-01T13:31:39.597663+0000","last_clean":"2024-04-01T13:31:39.597663+0000","last_became_active":"2024-04-01T13:30:45.693590+0000","last_became_peered":"2024-04-01T13:30:45.693590+0000","last_unstale":"2024-04-01T13:31:39.597663+0000","last_undegraded":"2024-04-01T13:31:39.597663+0000","last_fullsized":"2024-04-01T13:31:39.597663+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.643299+0000","last_change":"2024-04-01T13:30:24.495962+0000","last_active":"2024-04-01T13:30:51.643299+0000","last_peered":"2024-04-01T13:30:51.643299+0000","last_clean":"2024-04-01T13:30:51.643299+0000","last_became_active":"2024-04-01T13:30:24.489422+0000","last_became_peered":"2024-04-01T13:30:24.489422+0000","last_unstale":"2024-04-01T13:30:51.643299+0000","last_undegraded":"2024-04-01T13:30:51.643299+0000","last_fullsized":"2024-04-01T13:30:51.643299+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":98571801,"num_objects":42,"num_object_clones":4,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":183,"num_read_kb":73219,"num_write":239,"num_write_kb":66094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76828,"kb_used_data":49812,"kb_used_omap":0,"kb_used_meta":27008,"kb_avail":94295012,"statfs":{"total":96636764160,"available":96558092288,"internally_reserved":0,"allocated":51007488,"data_stored":51414745,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27656192},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64588,"kb_used_data":37764,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94307252,"statfs":{"total":96636764160,"available":96570626048,"internally_reserved":0,"allocated":38670336,"data_stored":39492342,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69828,"kb_used_data":42448,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94302012,"statfs":{"total":96636764160,"available":96565260288,"internally_reserved":0,"allocated":43466752,"data_stored":44221816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":29,"seq":124554051599,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71680,"kb_used_data":44728,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94300160,"statfs":{"total":96636764160,"available":96563363840,"internally_reserved":0,"allocated":45801472,"data_stored":47929376,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1508,"internal_metadata":27589148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-01T13:31:48.364 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-01T13:31:48.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:48.484+0000 7fa721ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:48.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:48.491+0000 7fa721ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:48.744 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:31:48.744 INFO:teuthology.orchestra.run.smithi059.stdout:{"fsid":"2a626e12-75d1-4fa9-b59b-fc64723c30fe","health":{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 3 pgs peering","count":3},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":95,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1711978289,"num_in_osds":4,"osd_in_since":1711978215,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"peering","count":6},{"state_name":"active+clean+snaptrim","count":2},{"state_name":"peering+snaptrim","count":1}],"num_pgs":25,"num_pools":3,"num_objects":44,"data_bytes":99162169,"bytes_used":289714176,"bytes_avail":386257342464,"bytes_total":386547056640,"inactive_pgs_ratio":0.2800000011920929},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-01T13:31:19.377894+0000","services":{}},"progress_events":{}} 2024-04-01T13:31:48.756 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-01T13:31:48.756 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T13:31:48.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:48.879+0000 7f5228154640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:48.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:48.887+0000 7f5228154640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:49.068 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:31:49.068 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-01T13:31:49.080 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-04-01T13:31:47.385078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99162169,"num_objects":44,"num_object_clones":4,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":286,"num_read_kb":73310,"num_write":376,"num_write_kb":67988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":337,"ondisk_log_size":337,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":282924,"kb_used_data":174752,"kb_used_omap":1,"kb_used_meta":107582,"kb_avail":377204436,"statfs":{"total":386547056640,"available":386257342464,"internally_reserved":0,"allocated":178946048,"data_stored":183058279,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":1508,"internal_metadata":110164508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.820633"},"pg_stats":[{"pgid":"3.f","version":"16'9","reported_seq":47,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.610970+0000","last_change":"2024-04-01T13:31:29.607744+0000","last_active":"2024-04-01T13:31:29.607744+0000","last_peered":"2024-04-01T13:30:45.695648+0000","last_clean":"2024-04-01T13:30:45.695648+0000","last_became_active":"2024-04-01T13:30:45.686209+0000","last_became_peered":"2024-04-01T13:30:45.686209+0000","last_unstale":"2024-04-01T13:31:38.610970+0000","last_undegraded":"2024-04-01T13:31:38.610970+0000","last_fullsized":"2024-04-01T13:31:38.610970+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7008420,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":4853,"num_write":12,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"26'11","reported_seq":76,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.599421+0000","last_change":"2024-04-01T13:30:53.709220+0000","last_active":"2024-04-01T13:31:39.599421+0000","last_peered":"2024-04-01T13:31:39.599421+0000","last_clean":"2024-04-01T13:31:39.599421+0000","last_became_active":"2024-04-01T13:30:45.689067+0000","last_became_peered":"2024-04-01T13:30:45.689067+0000","last_unstale":"2024-04-01T13:31:39.599421+0000","last_undegraded":"2024-04-01T13:31:39.599421+0000","last_fullsized":"2024-04-01T13:31:39.599421+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042034,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"25'23","reported_seq":81,"reported_epoch":26,"state":"active+clean+snaptrim","last_fresh":"2024-04-01T13:30:51.638760+0000","last_change":"2024-04-01T13:30:50.616859+0000","last_active":"2024-04-01T13:30:51.638760+0000","last_peered":"2024-04-01T13:30:51.638760+0000","last_clean":"2024-04-01T13:30:51.638760+0000","last_became_active":"2024-04-01T13:30:45.692470+0000","last_became_peered":"2024-04-01T13:30:45.692470+0000","last_unstale":"2024-04-01T13:30:51.638760+0000","last_undegraded":"2024-04-01T13:30:51.638760+0000","last_fullsized":"2024-04-01T13:30:51.638760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":10722734,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":5638,"num_write":26,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"28'26","reported_seq":86,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.618774+0000","last_change":"2024-04-01T13:31:36.614767+0000","last_active":"2024-04-01T13:31:39.618774+0000","last_peered":"2024-04-01T13:31:39.618774+0000","last_clean":"2024-04-01T13:31:39.618774+0000","last_became_active":"2024-04-01T13:31:30.574937+0000","last_became_peered":"2024-04-01T13:31:30.574937+0000","last_unstale":"2024-04-01T13:31:39.618774+0000","last_undegraded":"2024-04-01T13:31:39.618774+0000","last_fullsized":"2024-04-01T13:31:39.618774+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"16'12","reported_seq":89,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.638600+0000","last_change":"2024-04-01T13:31:36.596629+0000","last_active":"2024-04-01T13:31:39.638600+0000","last_peered":"2024-04-01T13:31:39.638600+0000","last_clean":"2024-04-01T13:31:39.638600+0000","last_became_active":"2024-04-01T13:31:30.604705+0000","last_became_peered":"2024-04-01T13:31:30.604705+0000","last_unstale":"2024-04-01T13:31:39.638600+0000","last_undegraded":"2024-04-01T13:31:39.638600+0000","last_fullsized":"2024-04-01T13:31:39.638600+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":10843,"num_write":16,"num_write_kb":4752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"24'11","reported_seq":60,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.624072+0000","last_change":"2024-04-01T13:31:29.600971+0000","last_active":"2024-04-01T13:31:29.600971+0000","last_peered":"2024-04-01T13:30:49.664233+0000","last_clean":"2024-04-01T13:30:49.664233+0000","last_became_active":"2024-04-01T13:30:30.656628+0000","last_became_peered":"2024-04-01T13:30:30.656628+0000","last_unstale":"2024-04-01T13:31:38.624072+0000","last_undegraded":"2024-04-01T13:31:38.624072+0000","last_fullsized":"2024-04-01T13:31:38.624072+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7495823,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10286,"num_write":13,"num_write_kb":3933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'20","reported_seq":60,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.617062+0000","last_change":"2024-04-01T13:31:29.542796+0000","last_active":"2024-04-01T13:31:29.542796+0000","last_peered":"2024-04-01T13:30:49.030107+0000","last_clean":"2024-04-01T13:30:49.030107+0000","last_became_active":"2024-04-01T13:30:30.635917+0000","last_became_peered":"2024-04-01T13:30:30.635917+0000","last_unstale":"2024-04-01T13:31:38.617062+0000","last_undegraded":"2024-04-01T13:31:38.617062+0000","last_fullsized":"2024-04-01T13:31:38.617062+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5865577,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":22,"num_write_kb":6047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"24'20","reported_seq":78,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.697588+0000","last_change":"2024-04-01T13:30:51.647095+0000","last_active":"2024-04-01T13:30:51.697588+0000","last_peered":"2024-04-01T13:30:51.697588+0000","last_clean":"2024-04-01T13:30:51.697588+0000","last_became_active":"2024-04-01T13:30:45.695964+0000","last_became_peered":"2024-04-01T13:30:45.695964+0000","last_unstale":"2024-04-01T13:30:51.697588+0000","last_undegraded":"2024-04-01T13:30:51.697588+0000","last_fullsized":"2024-04-01T13:30:51.697588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11243751,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":14764,"num_write":26,"num_write_kb":7239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":189,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.627624+0000","last_change":"2024-04-01T13:31:30.585158+0000","last_active":"2024-04-01T13:31:39.627624+0000","last_peered":"2024-04-01T13:31:39.627624+0000","last_clean":"2024-04-01T13:31:39.627624+0000","last_became_active":"2024-04-01T13:31:30.578315+0000","last_became_peered":"2024-04-01T13:31:30.578315+0000","last_unstale":"2024-04-01T13:31:39.627624+0000","last_undegraded":"2024-04-01T13:31:39.627624+0000","last_fullsized":"2024-04-01T13:31:39.627624+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_clean_scrub_stamp":"2024-04-01T13:30:22.378617+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.643211+0000","last_change":"2024-04-01T13:30:24.493976+0000","last_active":"2024-04-01T13:30:51.643211+0000","last_peered":"2024-04-01T13:30:51.643211+0000","last_clean":"2024-04-01T13:30:51.643211+0000","last_became_active":"2024-04-01T13:30:24.483528+0000","last_became_peered":"2024-04-01T13:30:24.483528+0000","last_unstale":"2024-04-01T13:30:51.643211+0000","last_undegraded":"2024-04-01T13:30:51.643211+0000","last_fullsized":"2024-04-01T13:30:51.643211+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"24'6","reported_seq":50,"reported_epoch":26,"state":"active+clean+snaptrim","last_fresh":"2024-04-01T13:30:51.640989+0000","last_change":"2024-04-01T13:30:50.619384+0000","last_active":"2024-04-01T13:30:51.640989+0000","last_peered":"2024-04-01T13:30:51.640989+0000","last_clean":"2024-04-01T13:30:51.640989+0000","last_became_active":"2024-04-01T13:30:45.703077+0000","last_became_peered":"2024-04-01T13:30:45.703077+0000","last_unstale":"2024-04-01T13:30:51.640989+0000","last_undegraded":"2024-04-01T13:30:51.640989+0000","last_fullsized":"2024-04-01T13:30:51.640989+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2352947,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.635036+0000","last_change":"2024-04-01T13:31:29.731657+0000","last_active":"2024-04-01T13:31:29.731657+0000","last_peered":"2024-04-01T13:30:24.494919+0000","last_clean":"2024-04-01T13:30:24.494919+0000","last_became_active":"2024-04-01T13:30:24.486637+0000","last_became_peered":"2024-04-01T13:30:24.486637+0000","last_unstale":"2024-04-01T13:31:38.635036+0000","last_undegraded":"2024-04-01T13:31:38.635036+0000","last_fullsized":"2024-04-01T13:31:38.635036+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"26'13","reported_seq":68,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.631922+0000","last_change":"2024-04-01T13:31:29.552816+0000","last_active":"2024-04-01T13:31:20.179135+0000","last_peered":"2024-04-01T13:30:51.678933+0000","last_clean":"2024-04-01T13:30:51.678933+0000","last_became_active":"2024-04-01T13:30:45.719645+0000","last_became_peered":"2024-04-01T13:30:45.719645+0000","last_unstale":"2024-04-01T13:31:38.631922+0000","last_undegraded":"2024-04-01T13:31:38.631922+0000","last_fullsized":"2024-04-01T13:31:38.631922+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8050454,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10129,"num_write":17,"num_write_kb":4892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":52,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.629011+0000","last_change":"2024-04-01T13:30:45.679818+0000","last_active":"2024-04-01T13:30:51.629011+0000","last_peered":"2024-04-01T13:30:51.629011+0000","last_clean":"2024-04-01T13:30:51.629011+0000","last_became_active":"2024-04-01T13:30:45.674014+0000","last_became_peered":"2024-04-01T13:30:45.674014+0000","last_unstale":"2024-04-01T13:30:51.629011+0000","last_undegraded":"2024-04-01T13:30:51.629011+0000","last_fullsized":"2024-04-01T13:30:51.629011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"26'35","reported_seq":103,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.629586+0000","last_change":"2024-04-01T13:30:53.716341+0000","last_active":"2024-04-01T13:31:39.629586+0000","last_peered":"2024-04-01T13:31:39.629586+0000","last_clean":"2024-04-01T13:31:39.629586+0000","last_became_active":"2024-04-01T13:30:45.733223+0000","last_became_peered":"2024-04-01T13:30:45.733223+0000","last_unstale":"2024-04-01T13:31:39.629586+0000","last_undegraded":"2024-04-01T13:31:39.629586+0000","last_fullsized":"2024-04-01T13:31:39.629586+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":12168103,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":5339,"num_write":35,"num_write_kb":8781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.632357+0000","last_change":"2024-04-01T13:31:29.726707+0000","last_active":"2024-04-01T13:31:29.726707+0000","last_peered":"2024-04-01T13:30:24.480593+0000","last_clean":"2024-04-01T13:30:24.480593+0000","last_became_active":"2024-04-01T13:30:24.472751+0000","last_became_peered":"2024-04-01T13:30:24.472751+0000","last_unstale":"2024-04-01T13:31:38.632357+0000","last_undegraded":"2024-04-01T13:31:38.632357+0000","last_fullsized":"2024-04-01T13:31:38.632357+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"26'9","reported_seq":91,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.586336+0000","last_change":"2024-04-01T13:31:37.084731+0000","last_active":"2024-04-01T13:31:39.586336+0000","last_peered":"2024-04-01T13:31:39.586336+0000","last_clean":"2024-04-01T13:31:39.586336+0000","last_became_active":"2024-04-01T13:31:30.573263+0000","last_became_peered":"2024-04-01T13:31:30.573263+0000","last_unstale":"2024-04-01T13:31:39.586336+0000","last_undegraded":"2024-04-01T13:31:39.586336+0000","last_fullsized":"2024-04-01T13:31:39.586336+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"26'15","reported_seq":61,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.656955+0000","last_change":"2024-04-01T13:30:51.714578+0000","last_active":"2024-04-01T13:30:51.656955+0000","last_peered":"2024-04-01T13:30:51.656955+0000","last_clean":"2024-04-01T13:30:51.656955+0000","last_became_active":"2024-04-01T13:30:45.690525+0000","last_became_peered":"2024-04-01T13:30:45.690525+0000","last_unstale":"2024-04-01T13:30:51.656955+0000","last_undegraded":"2024-04-01T13:30:51.656955+0000","last_fullsized":"2024-04-01T13:30:51.656955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067228,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":13,"num_write_kb":2827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":74,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.593086+0000","last_change":"2024-04-01T13:31:30.584389+0000","last_active":"2024-04-01T13:31:39.593086+0000","last_peered":"2024-04-01T13:31:39.593086+0000","last_clean":"2024-04-01T13:31:39.593086+0000","last_became_active":"2024-04-01T13:31:30.579930+0000","last_became_peered":"2024-04-01T13:31:30.579930+0000","last_unstale":"2024-04-01T13:31:39.593086+0000","last_undegraded":"2024-04-01T13:31:39.593086+0000","last_fullsized":"2024-04-01T13:31:39.593086+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"16'6","reported_seq":84,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.594742+0000","last_change":"2024-04-01T13:31:37.090769+0000","last_active":"2024-04-01T13:31:39.594742+0000","last_peered":"2024-04-01T13:31:39.594742+0000","last_clean":"2024-04-01T13:31:39.594742+0000","last_became_active":"2024-04-01T13:31:30.581362+0000","last_became_peered":"2024-04-01T13:31:30.581362+0000","last_unstale":"2024-04-01T13:31:39.594742+0000","last_undegraded":"2024-04-01T13:31:39.594742+0000","last_fullsized":"2024-04-01T13:31:39.594742+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4907578,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5282,"num_write":8,"num_write_kb":2191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.639590+0000","last_change":"2024-04-01T13:31:30.608391+0000","last_active":"2024-04-01T13:31:39.639590+0000","last_peered":"2024-04-01T13:31:39.639590+0000","last_clean":"2024-04-01T13:31:39.639590+0000","last_became_active":"2024-04-01T13:31:30.601821+0000","last_became_peered":"2024-04-01T13:31:30.601821+0000","last_unstale":"2024-04-01T13:31:39.639590+0000","last_undegraded":"2024-04-01T13:31:39.639590+0000","last_fullsized":"2024-04-01T13:31:39.639590+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"24'24","reported_seq":76,"reported_epoch":34,"state":"peering+snaptrim","last_fresh":"2024-04-01T13:31:38.620596+0000","last_change":"2024-04-01T13:31:29.513840+0000","last_active":"2024-04-01T13:31:20.116167+0000","last_peered":"2024-04-01T13:30:51.627795+0000","last_clean":"2024-04-01T13:30:51.627795+0000","last_became_active":"2024-04-01T13:30:45.693544+0000","last_became_peered":"2024-04-01T13:30:45.693544+0000","last_unstale":"2024-04-01T13:31:38.620596+0000","last_undegraded":"2024-04-01T13:31:38.620596+0000","last_fullsized":"2024-04-01T13:31:38.620596+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739541,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":6072,"num_write":29,"num_write_kb":8248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":74,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.590528+0000","last_change":"2024-04-01T13:31:30.594953+0000","last_active":"2024-04-01T13:31:39.590528+0000","last_peered":"2024-04-01T13:31:39.590528+0000","last_clean":"2024-04-01T13:31:39.590528+0000","last_became_active":"2024-04-01T13:31:30.586687+0000","last_became_peered":"2024-04-01T13:31:30.586687+0000","last_unstale":"2024-04-01T13:31:39.590528+0000","last_undegraded":"2024-04-01T13:31:39.590528+0000","last_fullsized":"2024-04-01T13:31:39.590528+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":66,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.597663+0000","last_change":"2024-04-01T13:30:51.650292+0000","last_active":"2024-04-01T13:31:39.597663+0000","last_peered":"2024-04-01T13:31:39.597663+0000","last_clean":"2024-04-01T13:31:39.597663+0000","last_became_active":"2024-04-01T13:30:45.693590+0000","last_became_peered":"2024-04-01T13:30:45.693590+0000","last_unstale":"2024-04-01T13:31:39.597663+0000","last_undegraded":"2024-04-01T13:31:39.597663+0000","last_fullsized":"2024-04-01T13:31:39.597663+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-01T13:30:51.643299+0000","last_change":"2024-04-01T13:30:24.495962+0000","last_active":"2024-04-01T13:30:51.643299+0000","last_peered":"2024-04-01T13:30:51.643299+0000","last_clean":"2024-04-01T13:30:51.643299+0000","last_became_active":"2024-04-01T13:30:24.489422+0000","last_became_peered":"2024-04-01T13:30:24.489422+0000","last_unstale":"2024-04-01T13:30:51.643299+0000","last_undegraded":"2024-04-01T13:30:51.643299+0000","last_fullsized":"2024-04-01T13:30:51.643299+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":98571801,"num_objects":42,"num_object_clones":4,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":183,"num_read_kb":73219,"num_write":239,"num_write_kb":66094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76828,"kb_used_data":49812,"kb_used_omap":0,"kb_used_meta":27008,"kb_avail":94295012,"statfs":{"total":96636764160,"available":96558092288,"internally_reserved":0,"allocated":51007488,"data_stored":51414745,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27656192},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64588,"kb_used_data":37764,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94307252,"statfs":{"total":96636764160,"available":96570626048,"internally_reserved":0,"allocated":38670336,"data_stored":39492342,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69828,"kb_used_data":42448,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94302012,"statfs":{"total":96636764160,"available":96565260288,"internally_reserved":0,"allocated":43466752,"data_stored":44221816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":29,"seq":124554051599,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71680,"kb_used_data":44728,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94300160,"statfs":{"total":96636764160,"available":96563363840,"internally_reserved":0,"allocated":45801472,"data_stored":47929376,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1508,"internal_metadata":27589148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-01T13:31:50.129 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENXIO: osd down 2024-04-01T13:31:50.271 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-01T13:31:50.271 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T13:31:51.235 INFO:tasks.rados.rados.0.smithi059.stdout:89: finishing write tid 1 to smithi05931183-7 2024-04-01T13:31:51.300 INFO:tasks.rados.rados.0.smithi059.stdout:92: finishing write tid 1 to smithi05931183-45 2024-04-01T13:31:51.301 INFO:tasks.rados.rados.0.smithi059.stdout:89: finishing write tid 2 to smithi05931183-7 2024-04-01T13:31:51.311 INFO:tasks.rados.rados.0.smithi059.stdout:102: expect (ObjNum 2 snap 0 seq_num 2) 2024-04-01T13:31:51.400 INFO:tasks.rados.rados.0.smithi059.stdout:88: finishing write tid 3 to smithi05931183-6 2024-04-01T13:31:51.400 INFO:tasks.rados.rados.0.smithi059.stdout:92: finishing write tid 2 to smithi05931183-45 2024-04-01T13:31:51.400 INFO:tasks.rados.rados.0.smithi059.stdout:102: done (8 left) 2024-04-01T13:31:51.400 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 8 2024-04-01T13:31:51.400 INFO:tasks.rados.rados.0.smithi059.stdout:89: finishing write tid 3 to smithi05931183-7 2024-04-01T13:31:51.400 INFO:tasks.rados.rados.0.smithi059.stdout:89: finishing write tid 4 to smithi05931183-7 2024-04-01T13:31:51.401 INFO:tasks.rados.rados.0.smithi059.stdout:89: oid 7 updating version 0 to 13 2024-04-01T13:31:51.401 INFO:tasks.rados.rados.0.smithi059.stdout:89: oid 7 version 13 is already newer than 0 2024-04-01T13:31:51.401 INFO:tasks.rados.rados.0.smithi059.stdout:89: oid 7 version 13 is already newer than 12 2024-04-01T13:31:51.401 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 7 v 13 (ObjNum 67 snap 3 seq_num 67) dirty exists 2024-04-01T13:31:51.401 INFO:tasks.rados.rados.0.smithi059.stdout:89: left oid 7 (ObjNum 67 snap 3 seq_num 67) 2024-04-01T13:31:51.401 INFO:tasks.rados.rados.0.smithi059.stdout:92: finishing write tid 3 to smithi05931183-45 2024-04-01T13:31:51.401 INFO:tasks.rados.rados.0.smithi059.stdout:89: done (7 left) 2024-04-01T13:31:51.401 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 7 2024-04-01T13:31:51.401 INFO:tasks.rados.rados.0.smithi059.stdout:92: finishing write tid 4 to smithi05931183-45 2024-04-01T13:31:51.401 INFO:tasks.rados.rados.0.smithi059.stdout:92: oid 45 updating version 0 to 22 2024-04-01T13:31:51.401 INFO:tasks.rados.rados.0.smithi059.stdout:92: oid 45 updating version 22 to 23 2024-04-01T13:31:51.401 INFO:tasks.rados.rados.0.smithi059.stdout:92: oid 45 version 23 is already newer than 21 2024-04-01T13:31:51.401 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 45 v 23 (ObjNum 70 snap 3 seq_num 70) dirty exists 2024-04-01T13:31:51.401 INFO:tasks.rados.rados.0.smithi059.stdout:92: left oid 45 (ObjNum 70 snap 3 seq_num 70) 2024-04-01T13:31:51.401 INFO:tasks.rados.rados.0.smithi059.stdout:92: done (6 left) 2024-04-01T13:31:51.402 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 6 2024-04-01T13:31:51.402 INFO:tasks.rados.rados.0.smithi059.stdout:101: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-01T13:31:51.513 INFO:tasks.rados.rados.0.smithi059.stdout:100: finishing write tid 1 to smithi05931183-21 2024-04-01T13:31:51.513 INFO:tasks.rados.rados.0.smithi059.stdout:88: finishing write tid 4 to smithi05931183-6 2024-04-01T13:31:51.513 INFO:tasks.rados.rados.0.smithi059.stdout:88: finishing write tid 5 to smithi05931183-6 2024-04-01T13:31:51.513 INFO:tasks.rados.rados.0.smithi059.stdout:88: oid 6 updating version 0 to 22 2024-04-01T13:31:51.513 INFO:tasks.rados.rados.0.smithi059.stdout:88: oid 6 updating version 22 to 23 2024-04-01T13:31:51.513 INFO:tasks.rados.rados.0.smithi059.stdout:88: oid 6 updating version 23 to 25 2024-04-01T13:31:51.513 INFO:tasks.rados.rados.0.smithi059.stdout:88: oid 6 version 25 is already newer than 0 2024-04-01T13:31:51.513 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 6 v 25 (ObjNum 66 snap 3 seq_num 66) dirty exists 2024-04-01T13:31:51.513 INFO:tasks.rados.rados.0.smithi059.stdout:88: left oid 6 (ObjNum 66 snap 3 seq_num 66) 2024-04-01T13:31:51.513 INFO:tasks.rados.rados.0.smithi059.stdout:90: finishing write tid 2 to smithi05931183-22 2024-04-01T13:31:51.513 INFO:tasks.rados.rados.0.smithi059.stdout:88: done (5 left) 2024-04-01T13:31:51.514 INFO:tasks.rados.rados.0.smithi059.stdout:101: done (4 left) 2024-04-01T13:31:51.514 INFO:tasks.rados.rados.0.smithi059.stdout:104: done (3 left) 2024-04-01T13:31:51.514 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 3 2024-04-01T13:31:51.514 INFO:tasks.rados.rados.0.smithi059.stdout:105: expect (ObjNum 55 snap 0 seq_num 55) 2024-04-01T13:31:51.659 INFO:tasks.rados.rados.0.smithi059.stdout:90: finishing write tid 3 to smithi05931183-22 2024-04-01T13:31:51.659 INFO:tasks.rados.rados.0.smithi059.stdout:100: finishing write tid 2 to smithi05931183-21 2024-04-01T13:31:51.659 INFO:tasks.rados.rados.0.smithi059.stdout:90: finishing write tid 4 to smithi05931183-22 2024-04-01T13:31:51.660 INFO:tasks.rados.rados.0.smithi059.stdout:90: finishing write tid 5 to smithi05931183-22 2024-04-01T13:31:51.660 INFO:tasks.rados.rados.0.smithi059.stdout:90: oid 22 updating version 0 to 8 2024-04-01T13:31:51.660 INFO:tasks.rados.rados.0.smithi059.stdout:90: oid 22 version 8 is already newer than 0 2024-04-01T13:31:51.660 INFO:tasks.rados.rados.0.smithi059.stdout:90: oid 22 version 8 is already newer than 5 2024-04-01T13:31:51.660 INFO:tasks.rados.rados.0.smithi059.stdout:90: oid 22 version 8 is already newer than 8 2024-04-01T13:31:51.660 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 22 v 8 (ObjNum 68 snap 3 seq_num 68) dirty exists 2024-04-01T13:31:51.660 INFO:tasks.rados.rados.0.smithi059.stdout:90: left oid 22 (ObjNum 68 snap 3 seq_num 68) 2024-04-01T13:31:51.660 INFO:tasks.rados.rados.0.smithi059.stdout:100: finishing write tid 3 to smithi05931183-21 2024-04-01T13:31:51.660 INFO:tasks.rados.rados.0.smithi059.stdout:90: done (2 left) 2024-04-01T13:31:51.660 INFO:tasks.rados.rados.0.smithi059.stdout:105: done (1 left) 2024-04-01T13:31:51.660 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 1 2024-04-01T13:31:51.660 INFO:tasks.rados.rados.0.smithi059.stdout:100: finishing write tid 4 to smithi05931183-21 2024-04-01T13:31:51.660 INFO:tasks.rados.rados.0.smithi059.stdout:100: finishing write tid 5 to smithi05931183-21 2024-04-01T13:31:51.660 INFO:tasks.rados.rados.0.smithi059.stdout:100: oid 21 version 0 is already newer than 0 2024-04-01T13:31:51.660 INFO:tasks.rados.rados.0.smithi059.stdout:100: oid 21 updating version 0 to 15 2024-04-01T13:31:51.661 INFO:tasks.rados.rados.0.smithi059.stdout:100: oid 21 updating version 15 to 16 2024-04-01T13:31:51.661 INFO:tasks.rados.rados.0.smithi059.stdout:100: oid 21 version 16 is already newer than 14 2024-04-01T13:31:51.661 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 21 v 16 (ObjNum 74 snap 3 seq_num 74) dirty exists 2024-04-01T13:31:51.661 INFO:tasks.rados.rados.0.smithi059.stdout:100: left oid 21 (ObjNum 74 snap 3 seq_num 74) 2024-04-01T13:31:51.661 INFO:tasks.rados.rados.0.smithi059.stdout:100: done (0 left) 2024-04-01T13:31:52.082 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T13:31:52.140 INFO:tasks.rados.rados.0.smithi059.stdout:108: done (0 left) 2024-04-01T13:31:52.140 INFO:tasks.rados.rados.0.smithi059.stdout:109: delete oid 42 current snap is 4 2024-04-01T13:31:52.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:52.215+0000 7f2f5195d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:52.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:52.225+0000 7f2f5195d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:52.228 INFO:tasks.rados.rados.0.smithi059.stdout:109: done (0 left) 2024-04-01T13:31:52.228 INFO:tasks.rados.rados.0.smithi059.stdout:110: read oid 48 snap 3 2024-04-01T13:31:52.228 INFO:tasks.rados.rados.0.smithi059.stdout:110: expect deleted 2024-04-01T13:31:52.228 INFO:tasks.rados.rados.0.smithi059.stdout:111: snap_create 2024-04-01T13:31:52.229 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 1 2024-04-01T13:31:52.237 INFO:tasks.rados.rados.0.smithi059.stdout:110: done (0 left) 2024-04-01T13:31:52.272 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T13:31:52.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:52.395+0000 7f969359d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:52.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:52.401+0000 7f969359d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:52.411 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:31:52.411 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-01T13:31:52.414 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-01T13:31:52.424 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-04-01T13:31:52.153172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99162169,"num_objects":44,"num_object_clones":4,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":286,"num_read_kb":73310,"num_write":376,"num_write_kb":67988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":337,"ondisk_log_size":337,"up":73,"acting":73,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":282924,"kb_used_data":174752,"kb_used_omap":1,"kb_used_meta":107582,"kb_avail":377204436,"statfs":{"total":386547056640,"available":386257342464,"internally_reserved":0,"allocated":178946048,"data_stored":183058279,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":1508,"internal_metadata":110164508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.768601"},"pg_stats":[{"pgid":"3.f","version":"16'9","reported_seq":47,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.610970+0000","last_change":"2024-04-01T13:31:29.607744+0000","last_active":"2024-04-01T13:31:29.607744+0000","last_peered":"2024-04-01T13:30:45.695648+0000","last_clean":"2024-04-01T13:30:45.695648+0000","last_became_active":"2024-04-01T13:30:45.686209+0000","last_became_peered":"2024-04-01T13:30:45.686209+0000","last_unstale":"2024-04-01T13:31:38.610970+0000","last_undegraded":"2024-04-01T13:31:38.610970+0000","last_fullsized":"2024-04-01T13:31:38.610970+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7008420,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":4853,"num_write":12,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"26'11","reported_seq":77,"reported_epoch":35,"state":"peering","last_fresh":"2024-04-01T13:31:39.599913+0000","last_change":"2024-04-01T13:31:50.202208+0000","last_active":"2024-04-01T13:31:50.202208+0000","last_peered":"2024-04-01T13:31:39.599913+0000","last_clean":"2024-04-01T13:31:39.599913+0000","last_became_active":"2024-04-01T13:30:45.689067+0000","last_became_peered":"2024-04-01T13:30:45.689067+0000","last_unstale":"2024-04-01T13:31:39.599913+0000","last_undegraded":"2024-04-01T13:31:39.599913+0000","last_fullsized":"2024-04-01T13:31:39.599913+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042034,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"25'23","reported_seq":81,"reported_epoch":26,"state":"stale+active+clean+snaptrim","last_fresh":"2024-04-01T13:30:51.638760+0000","last_change":"2024-04-01T13:30:50.616859+0000","last_active":"2024-04-01T13:30:51.638760+0000","last_peered":"2024-04-01T13:30:51.638760+0000","last_clean":"2024-04-01T13:30:51.638760+0000","last_became_active":"2024-04-01T13:30:45.692470+0000","last_became_peered":"2024-04-01T13:30:45.692470+0000","last_unstale":"2024-04-01T13:31:50.141285+0000","last_undegraded":"2024-04-01T13:30:51.638760+0000","last_fullsized":"2024-04-01T13:30:51.638760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":10722734,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":5638,"num_write":26,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"28'26","reported_seq":86,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.618774+0000","last_change":"2024-04-01T13:31:36.614767+0000","last_active":"2024-04-01T13:31:39.618774+0000","last_peered":"2024-04-01T13:31:39.618774+0000","last_clean":"2024-04-01T13:31:39.618774+0000","last_became_active":"2024-04-01T13:31:30.574937+0000","last_became_peered":"2024-04-01T13:31:30.574937+0000","last_unstale":"2024-04-01T13:31:39.618774+0000","last_undegraded":"2024-04-01T13:31:39.618774+0000","last_fullsized":"2024-04-01T13:31:39.618774+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"16'12","reported_seq":89,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.638600+0000","last_change":"2024-04-01T13:31:36.596629+0000","last_active":"2024-04-01T13:31:39.638600+0000","last_peered":"2024-04-01T13:31:39.638600+0000","last_clean":"2024-04-01T13:31:39.638600+0000","last_became_active":"2024-04-01T13:31:30.604705+0000","last_became_peered":"2024-04-01T13:31:30.604705+0000","last_unstale":"2024-04-01T13:31:39.638600+0000","last_undegraded":"2024-04-01T13:31:39.638600+0000","last_fullsized":"2024-04-01T13:31:39.638600+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":10843,"num_write":16,"num_write_kb":4752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"24'11","reported_seq":60,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.624072+0000","last_change":"2024-04-01T13:31:29.600971+0000","last_active":"2024-04-01T13:31:29.600971+0000","last_peered":"2024-04-01T13:30:49.664233+0000","last_clean":"2024-04-01T13:30:49.664233+0000","last_became_active":"2024-04-01T13:30:30.656628+0000","last_became_peered":"2024-04-01T13:30:30.656628+0000","last_unstale":"2024-04-01T13:31:38.624072+0000","last_undegraded":"2024-04-01T13:31:38.624072+0000","last_fullsized":"2024-04-01T13:31:38.624072+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7495823,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10286,"num_write":13,"num_write_kb":3933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'20","reported_seq":60,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.617062+0000","last_change":"2024-04-01T13:31:29.542796+0000","last_active":"2024-04-01T13:31:29.542796+0000","last_peered":"2024-04-01T13:30:49.030107+0000","last_clean":"2024-04-01T13:30:49.030107+0000","last_became_active":"2024-04-01T13:30:30.635917+0000","last_became_peered":"2024-04-01T13:30:30.635917+0000","last_unstale":"2024-04-01T13:31:38.617062+0000","last_undegraded":"2024-04-01T13:31:38.617062+0000","last_fullsized":"2024-04-01T13:31:38.617062+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5865577,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":22,"num_write_kb":6047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"24'20","reported_seq":78,"reported_epoch":26,"state":"stale+active+clean","last_fresh":"2024-04-01T13:30:51.697588+0000","last_change":"2024-04-01T13:30:51.647095+0000","last_active":"2024-04-01T13:30:51.697588+0000","last_peered":"2024-04-01T13:30:51.697588+0000","last_clean":"2024-04-01T13:30:51.697588+0000","last_became_active":"2024-04-01T13:30:45.695964+0000","last_became_peered":"2024-04-01T13:30:45.695964+0000","last_unstale":"2024-04-01T13:31:50.141289+0000","last_undegraded":"2024-04-01T13:30:51.697588+0000","last_fullsized":"2024-04-01T13:30:51.697588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11243751,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":14764,"num_write":26,"num_write_kb":7239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":189,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.627624+0000","last_change":"2024-04-01T13:31:30.585158+0000","last_active":"2024-04-01T13:31:39.627624+0000","last_peered":"2024-04-01T13:31:39.627624+0000","last_clean":"2024-04-01T13:31:39.627624+0000","last_became_active":"2024-04-01T13:31:30.578315+0000","last_became_peered":"2024-04-01T13:31:30.578315+0000","last_unstale":"2024-04-01T13:31:39.627624+0000","last_undegraded":"2024-04-01T13:31:39.627624+0000","last_fullsized":"2024-04-01T13:31:39.627624+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_clean_scrub_stamp":"2024-04-01T13:30:22.378617+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":26,"state":"stale+active+clean","last_fresh":"2024-04-01T13:30:51.643211+0000","last_change":"2024-04-01T13:30:24.493976+0000","last_active":"2024-04-01T13:30:51.643211+0000","last_peered":"2024-04-01T13:30:51.643211+0000","last_clean":"2024-04-01T13:30:51.643211+0000","last_became_active":"2024-04-01T13:30:24.483528+0000","last_became_peered":"2024-04-01T13:30:24.483528+0000","last_unstale":"2024-04-01T13:31:50.141292+0000","last_undegraded":"2024-04-01T13:30:51.643211+0000","last_fullsized":"2024-04-01T13:30:51.643211+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"24'6","reported_seq":50,"reported_epoch":26,"state":"stale+active+clean+snaptrim","last_fresh":"2024-04-01T13:30:51.640989+0000","last_change":"2024-04-01T13:30:50.619384+0000","last_active":"2024-04-01T13:30:51.640989+0000","last_peered":"2024-04-01T13:30:51.640989+0000","last_clean":"2024-04-01T13:30:51.640989+0000","last_became_active":"2024-04-01T13:30:45.703077+0000","last_became_peered":"2024-04-01T13:30:45.703077+0000","last_unstale":"2024-04-01T13:31:50.141296+0000","last_undegraded":"2024-04-01T13:30:51.640989+0000","last_fullsized":"2024-04-01T13:30:51.640989+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2352947,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.635036+0000","last_change":"2024-04-01T13:31:29.731657+0000","last_active":"2024-04-01T13:31:29.731657+0000","last_peered":"2024-04-01T13:30:24.494919+0000","last_clean":"2024-04-01T13:30:24.494919+0000","last_became_active":"2024-04-01T13:30:24.486637+0000","last_became_peered":"2024-04-01T13:30:24.486637+0000","last_unstale":"2024-04-01T13:31:38.635036+0000","last_undegraded":"2024-04-01T13:31:38.635036+0000","last_fullsized":"2024-04-01T13:31:38.635036+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"26'13","reported_seq":68,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.631922+0000","last_change":"2024-04-01T13:31:29.552816+0000","last_active":"2024-04-01T13:31:20.179135+0000","last_peered":"2024-04-01T13:30:51.678933+0000","last_clean":"2024-04-01T13:30:51.678933+0000","last_became_active":"2024-04-01T13:30:45.719645+0000","last_became_peered":"2024-04-01T13:30:45.719645+0000","last_unstale":"2024-04-01T13:31:38.631922+0000","last_undegraded":"2024-04-01T13:31:38.631922+0000","last_fullsized":"2024-04-01T13:31:38.631922+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8050454,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10129,"num_write":17,"num_write_kb":4892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":52,"reported_epoch":26,"state":"stale+active+clean","last_fresh":"2024-04-01T13:30:51.629011+0000","last_change":"2024-04-01T13:30:45.679818+0000","last_active":"2024-04-01T13:30:51.629011+0000","last_peered":"2024-04-01T13:30:51.629011+0000","last_clean":"2024-04-01T13:30:51.629011+0000","last_became_active":"2024-04-01T13:30:45.674014+0000","last_became_peered":"2024-04-01T13:30:45.674014+0000","last_unstale":"2024-04-01T13:31:50.141299+0000","last_undegraded":"2024-04-01T13:30:51.629011+0000","last_fullsized":"2024-04-01T13:30:51.629011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"26'35","reported_seq":103,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.629586+0000","last_change":"2024-04-01T13:30:53.716341+0000","last_active":"2024-04-01T13:31:39.629586+0000","last_peered":"2024-04-01T13:31:39.629586+0000","last_clean":"2024-04-01T13:31:39.629586+0000","last_became_active":"2024-04-01T13:30:45.733223+0000","last_became_peered":"2024-04-01T13:30:45.733223+0000","last_unstale":"2024-04-01T13:31:39.629586+0000","last_undegraded":"2024-04-01T13:31:39.629586+0000","last_fullsized":"2024-04-01T13:31:39.629586+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":12168103,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":5339,"num_write":35,"num_write_kb":8781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.632357+0000","last_change":"2024-04-01T13:31:29.726707+0000","last_active":"2024-04-01T13:31:29.726707+0000","last_peered":"2024-04-01T13:30:24.480593+0000","last_clean":"2024-04-01T13:30:24.480593+0000","last_became_active":"2024-04-01T13:30:24.472751+0000","last_became_peered":"2024-04-01T13:30:24.472751+0000","last_unstale":"2024-04-01T13:31:38.632357+0000","last_undegraded":"2024-04-01T13:31:38.632357+0000","last_fullsized":"2024-04-01T13:31:38.632357+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"26'9","reported_seq":94,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-01T13:31:50.164160+0000","last_change":"2024-04-01T13:31:37.084731+0000","last_active":"2024-04-01T13:31:50.164160+0000","last_peered":"2024-04-01T13:31:50.164160+0000","last_clean":"2024-04-01T13:31:50.164160+0000","last_became_active":"2024-04-01T13:31:30.573263+0000","last_became_peered":"2024-04-01T13:31:30.573263+0000","last_unstale":"2024-04-01T13:31:50.164160+0000","last_undegraded":"2024-04-01T13:31:50.164160+0000","last_fullsized":"2024-04-01T13:31:50.164160+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"26'15","reported_seq":61,"reported_epoch":26,"state":"stale+active+clean","last_fresh":"2024-04-01T13:30:51.656955+0000","last_change":"2024-04-01T13:30:51.714578+0000","last_active":"2024-04-01T13:30:51.656955+0000","last_peered":"2024-04-01T13:30:51.656955+0000","last_clean":"2024-04-01T13:30:51.656955+0000","last_became_active":"2024-04-01T13:30:45.690525+0000","last_became_peered":"2024-04-01T13:30:45.690525+0000","last_unstale":"2024-04-01T13:31:50.141302+0000","last_undegraded":"2024-04-01T13:30:51.656955+0000","last_fullsized":"2024-04-01T13:30:51.656955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067228,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":13,"num_write_kb":2827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":77,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-01T13:31:50.191158+0000","last_change":"2024-04-01T13:31:30.584389+0000","last_active":"2024-04-01T13:31:50.191158+0000","last_peered":"2024-04-01T13:31:50.191158+0000","last_clean":"2024-04-01T13:31:50.191158+0000","last_became_active":"2024-04-01T13:31:30.579930+0000","last_became_peered":"2024-04-01T13:31:30.579930+0000","last_unstale":"2024-04-01T13:31:50.191158+0000","last_undegraded":"2024-04-01T13:31:50.191158+0000","last_fullsized":"2024-04-01T13:31:50.191158+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"16'6","reported_seq":87,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-01T13:31:50.184341+0000","last_change":"2024-04-01T13:31:37.090769+0000","last_active":"2024-04-01T13:31:50.184341+0000","last_peered":"2024-04-01T13:31:50.184341+0000","last_clean":"2024-04-01T13:31:50.184341+0000","last_became_active":"2024-04-01T13:31:30.581362+0000","last_became_peered":"2024-04-01T13:31:30.581362+0000","last_unstale":"2024-04-01T13:31:50.184341+0000","last_undegraded":"2024-04-01T13:31:50.184341+0000","last_fullsized":"2024-04-01T13:31:50.184341+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4907578,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5282,"num_write":8,"num_write_kb":2191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.639590+0000","last_change":"2024-04-01T13:31:30.608391+0000","last_active":"2024-04-01T13:31:39.639590+0000","last_peered":"2024-04-01T13:31:39.639590+0000","last_clean":"2024-04-01T13:31:39.639590+0000","last_became_active":"2024-04-01T13:31:30.601821+0000","last_became_peered":"2024-04-01T13:31:30.601821+0000","last_unstale":"2024-04-01T13:31:39.639590+0000","last_undegraded":"2024-04-01T13:31:39.639590+0000","last_fullsized":"2024-04-01T13:31:39.639590+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"24'24","reported_seq":76,"reported_epoch":34,"state":"peering+snaptrim","last_fresh":"2024-04-01T13:31:38.620596+0000","last_change":"2024-04-01T13:31:29.513840+0000","last_active":"2024-04-01T13:31:20.116167+0000","last_peered":"2024-04-01T13:30:51.627795+0000","last_clean":"2024-04-01T13:30:51.627795+0000","last_became_active":"2024-04-01T13:30:45.693544+0000","last_became_peered":"2024-04-01T13:30:45.693544+0000","last_unstale":"2024-04-01T13:31:38.620596+0000","last_undegraded":"2024-04-01T13:31:38.620596+0000","last_fullsized":"2024-04-01T13:31:38.620596+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739541,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":6072,"num_write":29,"num_write_kb":8248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":77,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-01T13:31:50.189658+0000","last_change":"2024-04-01T13:31:30.594953+0000","last_active":"2024-04-01T13:31:50.189658+0000","last_peered":"2024-04-01T13:31:50.189658+0000","last_clean":"2024-04-01T13:31:50.189658+0000","last_became_active":"2024-04-01T13:31:30.586687+0000","last_became_peered":"2024-04-01T13:31:30.586687+0000","last_unstale":"2024-04-01T13:31:50.189658+0000","last_undegraded":"2024-04-01T13:31:50.189658+0000","last_fullsized":"2024-04-01T13:31:50.189658+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":67,"reported_epoch":35,"state":"peering","last_fresh":"2024-04-01T13:31:39.598475+0000","last_change":"2024-04-01T13:31:50.211754+0000","last_active":"2024-04-01T13:31:50.211754+0000","last_peered":"2024-04-01T13:31:39.598475+0000","last_clean":"2024-04-01T13:31:39.598475+0000","last_became_active":"2024-04-01T13:30:45.693590+0000","last_became_peered":"2024-04-01T13:30:45.693590+0000","last_unstale":"2024-04-01T13:31:39.598475+0000","last_undegraded":"2024-04-01T13:31:39.598475+0000","last_fullsized":"2024-04-01T13:31:39.598475+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":26,"state":"stale+active+clean","last_fresh":"2024-04-01T13:30:51.643299+0000","last_change":"2024-04-01T13:30:24.495962+0000","last_active":"2024-04-01T13:30:51.643299+0000","last_peered":"2024-04-01T13:30:51.643299+0000","last_clean":"2024-04-01T13:30:51.643299+0000","last_became_active":"2024-04-01T13:30:24.489422+0000","last_became_peered":"2024-04-01T13:30:24.489422+0000","last_unstale":"2024-04-01T13:31:50.141305+0000","last_undegraded":"2024-04-01T13:30:51.643299+0000","last_fullsized":"2024-04-01T13:30:51.643299+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":98571801,"num_objects":42,"num_object_clones":4,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":183,"num_read_kb":73219,"num_write":239,"num_write_kb":66094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":46,"acting":46,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76828,"kb_used_data":49812,"kb_used_omap":0,"kb_used_meta":27008,"kb_avail":94295012,"statfs":{"total":96636764160,"available":96558092288,"internally_reserved":0,"allocated":51007488,"data_stored":51414745,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27656192},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64588,"kb_used_data":37764,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94307252,"statfs":{"total":96636764160,"available":96570626048,"internally_reserved":0,"allocated":38670336,"data_stored":39492342,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69828,"kb_used_data":42448,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94302012,"statfs":{"total":96636764160,"available":96565260288,"internally_reserved":0,"allocated":43466752,"data_stored":44221816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":29,"seq":124554051604,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71680,"kb_used_data":44728,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94300160,"statfs":{"total":96636764160,"available":96563363840,"internally_reserved":0,"allocated":45801472,"data_stored":47929376,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1508,"internal_metadata":27589148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-01T13:31:52.425 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-01T13:31:52.457 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-01T13:31:52.457 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T13:31:52.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:52.549+0000 7f05751f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:52.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:52.557+0000 7f05751f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:52.808 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:31:52.808 INFO:teuthology.orchestra.run.smithi059.stdout:{"fsid":"2a626e12-75d1-4fa9-b59b-fc64723c30fe","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 5 pgs peering","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":99,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":3,"osd_up_since":1711978310,"num_in_osds":4,"osd_in_since":1711978215,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"peering","count":8},{"state_name":"stale+active+clean","count":5},{"state_name":"stale+active+clean+snaptrim","count":2},{"state_name":"peering+snaptrim","count":1}],"num_pgs":25,"num_pools":3,"num_objects":44,"data_bytes":99162169,"bytes_used":289714176,"bytes_avail":386257342464,"bytes_total":386547056640,"inactive_pgs_ratio":0.36000001430511475},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-01T13:31:19.377894+0000","services":{}},"progress_events":{}} 2024-04-01T13:31:52.821 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-01T13:31:52.821 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T13:31:52.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:52.943+0000 7ff8e0824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:52.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:52.952+0000 7ff8e0824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:53.134 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:31:53.135 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-01T13:31:53.138 INFO:tasks.rados.rados.0.smithi059.stdout:111: done (0 left) 2024-04-01T13:31:53.138 INFO:tasks.rados.rados.0.smithi059.stdout:112: snap_create 2024-04-01T13:31:53.147 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-04-01T13:31:52.153172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99162169,"num_objects":44,"num_object_clones":4,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":286,"num_read_kb":73310,"num_write":376,"num_write_kb":67988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":337,"ondisk_log_size":337,"up":73,"acting":73,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":282924,"kb_used_data":174752,"kb_used_omap":1,"kb_used_meta":107582,"kb_avail":377204436,"statfs":{"total":386547056640,"available":386257342464,"internally_reserved":0,"allocated":178946048,"data_stored":183058279,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":1508,"internal_metadata":110164508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.768601"},"pg_stats":[{"pgid":"3.f","version":"16'9","reported_seq":47,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.610970+0000","last_change":"2024-04-01T13:31:29.607744+0000","last_active":"2024-04-01T13:31:29.607744+0000","last_peered":"2024-04-01T13:30:45.695648+0000","last_clean":"2024-04-01T13:30:45.695648+0000","last_became_active":"2024-04-01T13:30:45.686209+0000","last_became_peered":"2024-04-01T13:30:45.686209+0000","last_unstale":"2024-04-01T13:31:38.610970+0000","last_undegraded":"2024-04-01T13:31:38.610970+0000","last_fullsized":"2024-04-01T13:31:38.610970+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7008420,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":4853,"num_write":12,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"26'11","reported_seq":77,"reported_epoch":35,"state":"peering","last_fresh":"2024-04-01T13:31:39.599913+0000","last_change":"2024-04-01T13:31:50.202208+0000","last_active":"2024-04-01T13:31:50.202208+0000","last_peered":"2024-04-01T13:31:39.599913+0000","last_clean":"2024-04-01T13:31:39.599913+0000","last_became_active":"2024-04-01T13:30:45.689067+0000","last_became_peered":"2024-04-01T13:30:45.689067+0000","last_unstale":"2024-04-01T13:31:39.599913+0000","last_undegraded":"2024-04-01T13:31:39.599913+0000","last_fullsized":"2024-04-01T13:31:39.599913+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042034,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"25'23","reported_seq":81,"reported_epoch":26,"state":"stale+active+clean+snaptrim","last_fresh":"2024-04-01T13:30:51.638760+0000","last_change":"2024-04-01T13:30:50.616859+0000","last_active":"2024-04-01T13:30:51.638760+0000","last_peered":"2024-04-01T13:30:51.638760+0000","last_clean":"2024-04-01T13:30:51.638760+0000","last_became_active":"2024-04-01T13:30:45.692470+0000","last_became_peered":"2024-04-01T13:30:45.692470+0000","last_unstale":"2024-04-01T13:31:50.141285+0000","last_undegraded":"2024-04-01T13:30:51.638760+0000","last_fullsized":"2024-04-01T13:30:51.638760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":10722734,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":5638,"num_write":26,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"28'26","reported_seq":86,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.618774+0000","last_change":"2024-04-01T13:31:36.614767+0000","last_active":"2024-04-01T13:31:39.618774+0000","last_peered":"2024-04-01T13:31:39.618774+0000","last_clean":"2024-04-01T13:31:39.618774+0000","last_became_active":"2024-04-01T13:31:30.574937+0000","last_became_peered":"2024-04-01T13:31:30.574937+0000","last_unstale":"2024-04-01T13:31:39.618774+0000","last_undegraded":"2024-04-01T13:31:39.618774+0000","last_fullsized":"2024-04-01T13:31:39.618774+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"16'12","reported_seq":89,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.638600+0000","last_change":"2024-04-01T13:31:36.596629+0000","last_active":"2024-04-01T13:31:39.638600+0000","last_peered":"2024-04-01T13:31:39.638600+0000","last_clean":"2024-04-01T13:31:39.638600+0000","last_became_active":"2024-04-01T13:31:30.604705+0000","last_became_peered":"2024-04-01T13:31:30.604705+0000","last_unstale":"2024-04-01T13:31:39.638600+0000","last_undegraded":"2024-04-01T13:31:39.638600+0000","last_fullsized":"2024-04-01T13:31:39.638600+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":10843,"num_write":16,"num_write_kb":4752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"24'11","reported_seq":60,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.624072+0000","last_change":"2024-04-01T13:31:29.600971+0000","last_active":"2024-04-01T13:31:29.600971+0000","last_peered":"2024-04-01T13:30:49.664233+0000","last_clean":"2024-04-01T13:30:49.664233+0000","last_became_active":"2024-04-01T13:30:30.656628+0000","last_became_peered":"2024-04-01T13:30:30.656628+0000","last_unstale":"2024-04-01T13:31:38.624072+0000","last_undegraded":"2024-04-01T13:31:38.624072+0000","last_fullsized":"2024-04-01T13:31:38.624072+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7495823,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10286,"num_write":13,"num_write_kb":3933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'20","reported_seq":60,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.617062+0000","last_change":"2024-04-01T13:31:29.542796+0000","last_active":"2024-04-01T13:31:29.542796+0000","last_peered":"2024-04-01T13:30:49.030107+0000","last_clean":"2024-04-01T13:30:49.030107+0000","last_became_active":"2024-04-01T13:30:30.635917+0000","last_became_peered":"2024-04-01T13:30:30.635917+0000","last_unstale":"2024-04-01T13:31:38.617062+0000","last_undegraded":"2024-04-01T13:31:38.617062+0000","last_fullsized":"2024-04-01T13:31:38.617062+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5865577,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":22,"num_write_kb":6047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"24'20","reported_seq":78,"reported_epoch":26,"state":"stale+active+clean","last_fresh":"2024-04-01T13:30:51.697588+0000","last_change":"2024-04-01T13:30:51.647095+0000","last_active":"2024-04-01T13:30:51.697588+0000","last_peered":"2024-04-01T13:30:51.697588+0000","last_clean":"2024-04-01T13:30:51.697588+0000","last_became_active":"2024-04-01T13:30:45.695964+0000","last_became_peered":"2024-04-01T13:30:45.695964+0000","last_unstale":"2024-04-01T13:31:50.141289+0000","last_undegraded":"2024-04-01T13:30:51.697588+0000","last_fullsized":"2024-04-01T13:30:51.697588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11243751,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":14764,"num_write":26,"num_write_kb":7239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":189,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.627624+0000","last_change":"2024-04-01T13:31:30.585158+0000","last_active":"2024-04-01T13:31:39.627624+0000","last_peered":"2024-04-01T13:31:39.627624+0000","last_clean":"2024-04-01T13:31:39.627624+0000","last_became_active":"2024-04-01T13:31:30.578315+0000","last_became_peered":"2024-04-01T13:31:30.578315+0000","last_unstale":"2024-04-01T13:31:39.627624+0000","last_undegraded":"2024-04-01T13:31:39.627624+0000","last_fullsized":"2024-04-01T13:31:39.627624+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_clean_scrub_stamp":"2024-04-01T13:30:22.378617+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":26,"state":"stale+active+clean","last_fresh":"2024-04-01T13:30:51.643211+0000","last_change":"2024-04-01T13:30:24.493976+0000","last_active":"2024-04-01T13:30:51.643211+0000","last_peered":"2024-04-01T13:30:51.643211+0000","last_clean":"2024-04-01T13:30:51.643211+0000","last_became_active":"2024-04-01T13:30:24.483528+0000","last_became_peered":"2024-04-01T13:30:24.483528+0000","last_unstale":"2024-04-01T13:31:50.141292+0000","last_undegraded":"2024-04-01T13:30:51.643211+0000","last_fullsized":"2024-04-01T13:30:51.643211+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"24'6","reported_seq":50,"reported_epoch":26,"state":"stale+active+clean+snaptrim","last_fresh":"2024-04-01T13:30:51.640989+0000","last_change":"2024-04-01T13:30:50.619384+0000","last_active":"2024-04-01T13:30:51.640989+0000","last_peered":"2024-04-01T13:30:51.640989+0000","last_clean":"2024-04-01T13:30:51.640989+0000","last_became_active":"2024-04-01T13:30:45.703077+0000","last_became_peered":"2024-04-01T13:30:45.703077+0000","last_unstale":"2024-04-01T13:31:50.141296+0000","last_undegraded":"2024-04-01T13:30:51.640989+0000","last_fullsized":"2024-04-01T13:30:51.640989+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2352947,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.635036+0000","last_change":"2024-04-01T13:31:29.731657+0000","last_active":"2024-04-01T13:31:29.731657+0000","last_peered":"2024-04-01T13:30:24.494919+0000","last_clean":"2024-04-01T13:30:24.494919+0000","last_became_active":"2024-04-01T13:30:24.486637+0000","last_became_peered":"2024-04-01T13:30:24.486637+0000","last_unstale":"2024-04-01T13:31:38.635036+0000","last_undegraded":"2024-04-01T13:31:38.635036+0000","last_fullsized":"2024-04-01T13:31:38.635036+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"26'13","reported_seq":68,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.631922+0000","last_change":"2024-04-01T13:31:29.552816+0000","last_active":"2024-04-01T13:31:20.179135+0000","last_peered":"2024-04-01T13:30:51.678933+0000","last_clean":"2024-04-01T13:30:51.678933+0000","last_became_active":"2024-04-01T13:30:45.719645+0000","last_became_peered":"2024-04-01T13:30:45.719645+0000","last_unstale":"2024-04-01T13:31:38.631922+0000","last_undegraded":"2024-04-01T13:31:38.631922+0000","last_fullsized":"2024-04-01T13:31:38.631922+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8050454,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10129,"num_write":17,"num_write_kb":4892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":52,"reported_epoch":26,"state":"stale+active+clean","last_fresh":"2024-04-01T13:30:51.629011+0000","last_change":"2024-04-01T13:30:45.679818+0000","last_active":"2024-04-01T13:30:51.629011+0000","last_peered":"2024-04-01T13:30:51.629011+0000","last_clean":"2024-04-01T13:30:51.629011+0000","last_became_active":"2024-04-01T13:30:45.674014+0000","last_became_peered":"2024-04-01T13:30:45.674014+0000","last_unstale":"2024-04-01T13:31:50.141299+0000","last_undegraded":"2024-04-01T13:30:51.629011+0000","last_fullsized":"2024-04-01T13:30:51.629011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"26'35","reported_seq":103,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.629586+0000","last_change":"2024-04-01T13:30:53.716341+0000","last_active":"2024-04-01T13:31:39.629586+0000","last_peered":"2024-04-01T13:31:39.629586+0000","last_clean":"2024-04-01T13:31:39.629586+0000","last_became_active":"2024-04-01T13:30:45.733223+0000","last_became_peered":"2024-04-01T13:30:45.733223+0000","last_unstale":"2024-04-01T13:31:39.629586+0000","last_undegraded":"2024-04-01T13:31:39.629586+0000","last_fullsized":"2024-04-01T13:31:39.629586+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":12168103,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":5339,"num_write":35,"num_write_kb":8781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":34,"state":"peering","last_fresh":"2024-04-01T13:31:38.632357+0000","last_change":"2024-04-01T13:31:29.726707+0000","last_active":"2024-04-01T13:31:29.726707+0000","last_peered":"2024-04-01T13:30:24.480593+0000","last_clean":"2024-04-01T13:30:24.480593+0000","last_became_active":"2024-04-01T13:30:24.472751+0000","last_became_peered":"2024-04-01T13:30:24.472751+0000","last_unstale":"2024-04-01T13:31:38.632357+0000","last_undegraded":"2024-04-01T13:31:38.632357+0000","last_fullsized":"2024-04-01T13:31:38.632357+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"26'9","reported_seq":94,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-01T13:31:50.164160+0000","last_change":"2024-04-01T13:31:37.084731+0000","last_active":"2024-04-01T13:31:50.164160+0000","last_peered":"2024-04-01T13:31:50.164160+0000","last_clean":"2024-04-01T13:31:50.164160+0000","last_became_active":"2024-04-01T13:31:30.573263+0000","last_became_peered":"2024-04-01T13:31:30.573263+0000","last_unstale":"2024-04-01T13:31:50.164160+0000","last_undegraded":"2024-04-01T13:31:50.164160+0000","last_fullsized":"2024-04-01T13:31:50.164160+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"26'15","reported_seq":61,"reported_epoch":26,"state":"stale+active+clean","last_fresh":"2024-04-01T13:30:51.656955+0000","last_change":"2024-04-01T13:30:51.714578+0000","last_active":"2024-04-01T13:30:51.656955+0000","last_peered":"2024-04-01T13:30:51.656955+0000","last_clean":"2024-04-01T13:30:51.656955+0000","last_became_active":"2024-04-01T13:30:45.690525+0000","last_became_peered":"2024-04-01T13:30:45.690525+0000","last_unstale":"2024-04-01T13:31:50.141302+0000","last_undegraded":"2024-04-01T13:30:51.656955+0000","last_fullsized":"2024-04-01T13:30:51.656955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067228,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":13,"num_write_kb":2827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":77,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-01T13:31:50.191158+0000","last_change":"2024-04-01T13:31:30.584389+0000","last_active":"2024-04-01T13:31:50.191158+0000","last_peered":"2024-04-01T13:31:50.191158+0000","last_clean":"2024-04-01T13:31:50.191158+0000","last_became_active":"2024-04-01T13:31:30.579930+0000","last_became_peered":"2024-04-01T13:31:30.579930+0000","last_unstale":"2024-04-01T13:31:50.191158+0000","last_undegraded":"2024-04-01T13:31:50.191158+0000","last_fullsized":"2024-04-01T13:31:50.191158+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"16'6","reported_seq":87,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-01T13:31:50.184341+0000","last_change":"2024-04-01T13:31:37.090769+0000","last_active":"2024-04-01T13:31:50.184341+0000","last_peered":"2024-04-01T13:31:50.184341+0000","last_clean":"2024-04-01T13:31:50.184341+0000","last_became_active":"2024-04-01T13:31:30.581362+0000","last_became_peered":"2024-04-01T13:31:30.581362+0000","last_unstale":"2024-04-01T13:31:50.184341+0000","last_undegraded":"2024-04-01T13:31:50.184341+0000","last_fullsized":"2024-04-01T13:31:50.184341+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4907578,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5282,"num_write":8,"num_write_kb":2191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-01T13:31:39.639590+0000","last_change":"2024-04-01T13:31:30.608391+0000","last_active":"2024-04-01T13:31:39.639590+0000","last_peered":"2024-04-01T13:31:39.639590+0000","last_clean":"2024-04-01T13:31:39.639590+0000","last_became_active":"2024-04-01T13:31:30.601821+0000","last_became_peered":"2024-04-01T13:31:30.601821+0000","last_unstale":"2024-04-01T13:31:39.639590+0000","last_undegraded":"2024-04-01T13:31:39.639590+0000","last_fullsized":"2024-04-01T13:31:39.639590+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"24'24","reported_seq":76,"reported_epoch":34,"state":"peering+snaptrim","last_fresh":"2024-04-01T13:31:38.620596+0000","last_change":"2024-04-01T13:31:29.513840+0000","last_active":"2024-04-01T13:31:20.116167+0000","last_peered":"2024-04-01T13:30:51.627795+0000","last_clean":"2024-04-01T13:30:51.627795+0000","last_became_active":"2024-04-01T13:30:45.693544+0000","last_became_peered":"2024-04-01T13:30:45.693544+0000","last_unstale":"2024-04-01T13:31:38.620596+0000","last_undegraded":"2024-04-01T13:31:38.620596+0000","last_fullsized":"2024-04-01T13:31:38.620596+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739541,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":6072,"num_write":29,"num_write_kb":8248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":77,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-01T13:31:50.189658+0000","last_change":"2024-04-01T13:31:30.594953+0000","last_active":"2024-04-01T13:31:50.189658+0000","last_peered":"2024-04-01T13:31:50.189658+0000","last_clean":"2024-04-01T13:31:50.189658+0000","last_became_active":"2024-04-01T13:31:30.586687+0000","last_became_peered":"2024-04-01T13:31:30.586687+0000","last_unstale":"2024-04-01T13:31:50.189658+0000","last_undegraded":"2024-04-01T13:31:50.189658+0000","last_fullsized":"2024-04-01T13:31:50.189658+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":67,"reported_epoch":35,"state":"peering","last_fresh":"2024-04-01T13:31:39.598475+0000","last_change":"2024-04-01T13:31:50.211754+0000","last_active":"2024-04-01T13:31:50.211754+0000","last_peered":"2024-04-01T13:31:39.598475+0000","last_clean":"2024-04-01T13:31:39.598475+0000","last_became_active":"2024-04-01T13:30:45.693590+0000","last_became_peered":"2024-04-01T13:30:45.693590+0000","last_unstale":"2024-04-01T13:31:39.598475+0000","last_undegraded":"2024-04-01T13:31:39.598475+0000","last_fullsized":"2024-04-01T13:31:39.598475+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":26,"state":"stale+active+clean","last_fresh":"2024-04-01T13:30:51.643299+0000","last_change":"2024-04-01T13:30:24.495962+0000","last_active":"2024-04-01T13:30:51.643299+0000","last_peered":"2024-04-01T13:30:51.643299+0000","last_clean":"2024-04-01T13:30:51.643299+0000","last_became_active":"2024-04-01T13:30:24.489422+0000","last_became_peered":"2024-04-01T13:30:24.489422+0000","last_unstale":"2024-04-01T13:31:50.141305+0000","last_undegraded":"2024-04-01T13:30:51.643299+0000","last_fullsized":"2024-04-01T13:30:51.643299+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":98571801,"num_objects":42,"num_object_clones":4,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":183,"num_read_kb":73219,"num_write":239,"num_write_kb":66094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":46,"acting":46,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76828,"kb_used_data":49812,"kb_used_omap":0,"kb_used_meta":27008,"kb_avail":94295012,"statfs":{"total":96636764160,"available":96558092288,"internally_reserved":0,"allocated":51007488,"data_stored":51414745,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27656192},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64588,"kb_used_data":37764,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94307252,"statfs":{"total":96636764160,"available":96570626048,"internally_reserved":0,"allocated":38670336,"data_stored":39492342,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69828,"kb_used_data":42448,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94302012,"statfs":{"total":96636764160,"available":96565260288,"internally_reserved":0,"allocated":43466752,"data_stored":44221816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":29,"seq":124554051604,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71680,"kb_used_data":44728,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94300160,"statfs":{"total":96636764160,"available":96563363840,"internally_reserved":0,"allocated":45801472,"data_stored":47929376,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1508,"internal_metadata":27589148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-01T13:31:53.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~27s 2024-04-01T13:31:54.146 INFO:tasks.rados.rados.0.smithi059.stdout:112: done (0 left) 2024-04-01T13:31:54.146 INFO:tasks.rados.rados.0.smithi059.stdout:113: read oid 35 snap -1 2024-04-01T13:31:54.146 INFO:tasks.rados.rados.0.smithi059.stdout:113: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-01T13:31:54.146 INFO:tasks.rados.rados.0.smithi059.stdout:114: read oid 9 snap -1 2024-04-01T13:31:54.146 INFO:tasks.rados.rados.0.smithi059.stdout:114: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-01T13:31:54.146 INFO:tasks.rados.rados.0.smithi059.stdout:115: delete oid 26 current snap is 6 2024-04-01T13:31:54.239 INFO:tasks.rados.rados.0.smithi059.stdout:113: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-01T13:31:54.339 INFO:tasks.rados.rados.0.smithi059.stdout:114: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-01T13:31:54.422 INFO:tasks.rados.rados.0.smithi059.stdout:113: done (2 left) 2024-04-01T13:31:54.422 INFO:tasks.rados.rados.0.smithi059.stdout:114: done (1 left) 2024-04-01T13:31:54.423 INFO:tasks.rados.rados.0.smithi059.stdout:115: done (0 left) 2024-04-01T13:31:54.423 INFO:tasks.rados.rados.0.smithi059.stdout:116: write oid 28 current snap is 6 2024-04-01T13:31:54.423 INFO:tasks.rados.rados.0.smithi059.stdout:116: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2024-04-01T13:31:54.428 INFO:tasks.rados.rados.0.smithi059.stdout:116: writing smithi05931183-28 from 510946 to 1143898 tid 1 2024-04-01T13:31:54.434 INFO:tasks.rados.rados.0.smithi059.stdout:116: writing smithi05931183-28 from 1798066 to 2275745 tid 2 2024-04-01T13:31:54.439 INFO:tasks.rados.rados.0.smithi059.stdout:116: writing smithi05931183-28 from 2742758 to 3199975 tid 3 2024-04-01T13:31:54.440 INFO:tasks.rados.rados.0.smithi059.stdout:117: write oid 50 current snap is 6 2024-04-01T13:31:54.440 INFO:tasks.rados.rados.0.smithi059.stdout:117: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2024-04-01T13:31:54.446 INFO:tasks.rados.rados.0.smithi059.stdout:117: writing smithi05931183-50 from 527753 to 1235954 tid 1 2024-04-01T13:31:54.454 INFO:tasks.rados.rados.0.smithi059.stdout:117: writing smithi05931183-50 from 1856548 to 2594238 tid 2 2024-04-01T13:31:54.458 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T13:31:54.459 INFO:tasks.rados.rados.0.smithi059.stdout:117: writing smithi05931183-50 from 3286534 to 3327753 tid 3 2024-04-01T13:31:54.459 INFO:tasks.rados.rados.0.smithi059.stdout:118: write oid 49 current snap is 6 2024-04-01T13:31:54.459 INFO:tasks.rados.rados.0.smithi059.stdout:118: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-04-01T13:31:54.465 INFO:tasks.rados.rados.0.smithi059.stdout:118: writing smithi05931183-49 from 544560 to 1328010 tid 1 2024-04-01T13:31:54.471 INFO:tasks.rados.rados.0.smithi059.stdout:118: writing smithi05931183-49 from 1915030 to 2396378 tid 2 2024-04-01T13:31:54.477 INFO:tasks.rados.rados.0.smithi059.stdout:118: writing smithi05931183-49 from 2797604 to 3277365 tid 3 2024-04-01T13:31:54.477 INFO:tasks.rados.rados.0.smithi059.stdout:119: write (excl) oid 26 current snap is 6 2024-04-01T13:31:54.477 INFO:tasks.rados.rados.0.smithi059.stdout:119: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-04-01T13:31:54.485 INFO:tasks.rados.rados.0.smithi059.stdout:119: writing smithi05931183-26 from 561367 to 1020066 tid 1 2024-04-01T13:31:54.494 INFO:tasks.rados.rados.0.smithi059.stdout:119: writing smithi05931183-26 from 1573512 to 2314871 tid 2 2024-04-01T13:31:54.500 INFO:tasks.rados.rados.0.smithi059.stdout:119: writing smithi05931183-26 from 2941380 to 3361367 tid 3 2024-04-01T13:31:54.500 INFO:tasks.rados.rados.0.smithi059.stdout:120: write (excl) oid 44 current snap is 6 2024-04-01T13:31:54.500 INFO:tasks.rados.rados.0.smithi059.stdout:120: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2024-04-01T13:31:54.506 INFO:tasks.rados.rados.0.smithi059.stdout:120: writing smithi05931183-44 from 578174 to 1112122 tid 1 2024-04-01T13:31:54.511 INFO:tasks.rados.rados.0.smithi059.stdout:120: writing smithi05931183-44 from 1515641 to 2000658 tid 2 2024-04-01T13:31:54.519 INFO:tasks.rados.rados.0.smithi059.stdout:120: writing smithi05931183-44 from 2736097 to 3238402 tid 3 2024-04-01T13:31:54.519 INFO:tasks.rados.rados.0.smithi059.stdout:116: finishing write tid 1 to smithi05931183-28 2024-04-01T13:31:54.519 INFO:tasks.rados.rados.0.smithi059.stdout:121: read oid 22 snap -1 2024-04-01T13:31:54.519 INFO:tasks.rados.rados.0.smithi059.stdout:121: expect (ObjNum 68 snap 3 seq_num 68) 2024-04-01T13:31:54.519 INFO:tasks.rados.rados.0.smithi059.stdout:122: snap_remove snap 5 2024-04-01T13:31:54.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:54.588+0000 7f0486e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:54.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:54.596+0000 7f0486e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:54.642 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-01T13:31:54.710 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-01T13:31:54.710 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T13:31:55.148 INFO:tasks.rados.rados.0.smithi059.stdout:122: done (6 left) 2024-04-01T13:31:55.148 INFO:tasks.rados.rados.0.smithi059.stdout:123: read oid 37 snap -1 2024-04-01T13:31:55.148 INFO:tasks.rados.rados.0.smithi059.stdout:123: expect (ObjNum 36 snap 0 seq_num 36) 2024-04-01T13:31:55.148 INFO:tasks.rados.rados.0.smithi059.stdout:124: snap_remove snap 4 2024-04-01T13:31:56.149 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T13:31:56.150 INFO:tasks.rados.rados.0.smithi059.stdout:124: done (7 left) 2024-04-01T13:31:56.150 INFO:tasks.rados.rados.0.smithi059.stdout:125: snap_remove snap 3 2024-04-01T13:31:56.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:56.284+0000 7f67aa22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:56.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:56.292+0000 7f67aa22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:56.473 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:31:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-01T13:31:56.485 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-04-01T13:31:56.162634+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":85464662,"num_objects":36,"num_object_clones":2,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":277,"num_read_kb":75814,"num_write":344,"num_write_kb":58329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":678477,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":358,"ondisk_log_size":358,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":283564,"kb_used_data":175392,"kb_used_omap":1,"kb_used_meta":107582,"kb_avail":377203796,"statfs":{"total":386547056640,"available":386256687104,"internally_reserved":0,"allocated":179601408,"data_stored":181427300,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":1508,"internal_metadata":110164508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13697507,"num_objects":-8,"num_object_clones":-2,"num_object_copies":-24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":-9,"num_read_kb":2504,"num_write":-32,"num_write_kb":-9659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":-1123850,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.009462"},"pg_stats":[{"pgid":"3.f","version":"16'9","reported_seq":67,"reported_epoch":39,"state":"active+undersized+degraded","last_fresh":"2024-04-01T13:31:54.569341+0000","last_change":"2024-04-01T13:31:52.201632+0000","last_active":"2024-04-01T13:31:54.569341+0000","last_peered":"2024-04-01T13:31:54.569341+0000","last_clean":"2024-04-01T13:30:45.695648+0000","last_became_active":"2024-04-01T13:31:51.253156+0000","last_became_peered":"2024-04-01T13:31:51.253156+0000","last_unstale":"2024-04-01T13:31:54.569341+0000","last_undegraded":"2024-04-01T13:31:51.171139+0000","last_fullsized":"2024-04-01T13:31:51.169385+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7008420,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":9786,"num_write":12,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"26'11","reported_seq":97,"reported_epoch":40,"state":"active+undersized+degraded","last_fresh":"2024-04-01T13:31:55.193722+0000","last_change":"2024-04-01T13:31:55.204853+0000","last_active":"2024-04-01T13:31:55.193722+0000","last_peered":"2024-04-01T13:31:55.193722+0000","last_clean":"2024-04-01T13:31:39.599913+0000","last_became_active":"2024-04-01T13:31:51.231828+0000","last_became_peered":"2024-04-01T13:31:51.231828+0000","last_unstale":"2024-04-01T13:31:55.193722+0000","last_undegraded":"2024-04-01T13:31:51.173803+0000","last_fullsized":"2024-04-01T13:31:51.171147+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042034,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"25'23","reported_seq":67,"reported_epoch":38,"state":"active+undersized+degraded","last_fresh":"2024-04-01T13:31:53.184319+0000","last_change":"2024-04-01T13:31:52.209303+0000","last_active":"2024-04-01T13:31:53.184319+0000","last_peered":"2024-04-01T13:31:53.184319+0000","last_clean":"2024-04-01T13:30:34.592347+0000","last_became_active":"2024-04-01T13:31:51.268999+0000","last_became_peered":"2024-04-01T13:31:51.268999+0000","last_unstale":"2024-04-01T13:31:53.184319+0000","last_undegraded":"2024-04-01T13:31:51.189944+0000","last_fullsized":"2024-04-01T13:31:51.186639+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10369787,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":22,"num_write_kb":6296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"37'28","reported_seq":100,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-01T13:31:54.590154+0000","last_change":"2024-04-01T13:31:36.614767+0000","last_active":"2024-04-01T13:31:54.590154+0000","last_peered":"2024-04-01T13:31:54.590154+0000","last_clean":"2024-04-01T13:31:54.590154+0000","last_became_active":"2024-04-01T13:31:30.574937+0000","last_became_peered":"2024-04-01T13:31:30.574937+0000","last_unstale":"2024-04-01T13:31:54.590154+0000","last_undegraded":"2024-04-01T13:31:54.590154+0000","last_fullsized":"2024-04-01T13:31:54.590154+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"16'12","reported_seq":101,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-01T13:31:53.179276+0000","last_change":"2024-04-01T13:31:36.596629+0000","last_active":"2024-04-01T13:31:53.179276+0000","last_peered":"2024-04-01T13:31:53.179276+0000","last_clean":"2024-04-01T13:31:53.179276+0000","last_became_active":"2024-04-01T13:31:30.604705+0000","last_became_peered":"2024-04-01T13:31:30.604705+0000","last_unstale":"2024-04-01T13:31:53.179276+0000","last_undegraded":"2024-04-01T13:31:53.179276+0000","last_fullsized":"2024-04-01T13:31:53.179276+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":10843,"num_write":16,"num_write_kb":4752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"37'15","reported_seq":84,"reported_epoch":39,"state":"active+undersized+degraded","last_fresh":"2024-04-01T13:31:54.596903+0000","last_change":"2024-04-01T13:31:54.237314+0000","last_active":"2024-04-01T13:31:54.596903+0000","last_peered":"2024-04-01T13:31:54.596903+0000","last_clean":"2024-04-01T13:30:49.664233+0000","last_became_active":"2024-04-01T13:31:51.233044+0000","last_became_peered":"2024-04-01T13:31:51.233044+0000","last_unstale":"2024-04-01T13:31:54.596903+0000","last_undegraded":"2024-04-01T13:31:51.180750+0000","last_fullsized":"2024-04-01T13:31:51.179132+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":8521050,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":18414,"num_write":16,"num_write_kb":4632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"37'25","reported_seq":82,"reported_epoch":39,"state":"active+undersized+degraded","last_fresh":"2024-04-01T13:31:54.576161+0000","last_change":"2024-04-01T13:31:54.218951+0000","last_active":"2024-04-01T13:31:54.576161+0000","last_peered":"2024-04-01T13:31:54.576161+0000","last_clean":"2024-04-01T13:30:49.030107+0000","last_became_active":"2024-04-01T13:31:51.222200+0000","last_became_peered":"2024-04-01T13:31:51.222200+0000","last_unstale":"2024-04-01T13:31:54.576161+0000","last_undegraded":"2024-04-01T13:31:51.166334+0000","last_fullsized":"2024-04-01T13:31:51.164616+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":6016840,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":26,"num_write_kb":7311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"24'20","reported_seq":18,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-04-01T13:31:53.182739+0000","last_change":"2024-04-01T13:31:52.200143+0000","last_active":"2024-04-01T13:31:53.182739+0000","last_peered":"2024-04-01T13:31:53.182739+0000","last_clean":"2024-04-01T13:30:29.519276+0000","last_became_active":"2024-04-01T13:31:51.243345+0000","last_became_peered":"2024-04-01T13:31:51.243345+0000","last_unstale":"2024-04-01T13:31:53.182739+0000","last_undegraded":"2024-04-01T13:31:53.182739+0000","last_fullsized":"2024-04-01T13:31:51.177096+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":201,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-01T13:31:53.163276+0000","last_change":"2024-04-01T13:31:30.585158+0000","last_active":"2024-04-01T13:31:53.163276+0000","last_peered":"2024-04-01T13:31:53.163276+0000","last_clean":"2024-04-01T13:31:53.163276+0000","last_became_active":"2024-04-01T13:31:30.578315+0000","last_became_peered":"2024-04-01T13:31:30.578315+0000","last_unstale":"2024-04-01T13:31:53.163276+0000","last_undegraded":"2024-04-01T13:31:53.163276+0000","last_fullsized":"2024-04-01T13:31:53.163276+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_clean_scrub_stamp":"2024-04-01T13:30:22.378617+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-04-01T13:31:54.560497+0000","last_change":"2024-04-01T13:31:51.226808+0000","last_active":"2024-04-01T13:31:54.560497+0000","last_peered":"2024-04-01T13:31:54.560497+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:31:51.226808+0000","last_became_peered":"2024-04-01T13:31:51.226808+0000","last_unstale":"2024-04-01T13:31:54.560497+0000","last_undegraded":"2024-04-01T13:31:54.560497+0000","last_fullsized":"2024-04-01T13:31:51.158519+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"37'10","reported_seq":31,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-04-01T13:31:54.631203+0000","last_change":"2024-04-01T13:31:54.251574+0000","last_active":"2024-04-01T13:31:54.631203+0000","last_peered":"2024-04-01T13:31:54.631203+0000","last_clean":"2024-04-01T13:30:29.519276+0000","last_became_active":"2024-04-01T13:31:51.273685+0000","last_became_peered":"2024-04-01T13:31:51.273685+0000","last_unstale":"2024-04-01T13:31:54.631203+0000","last_undegraded":"2024-04-01T13:31:54.631203+0000","last_fullsized":"2024-04-01T13:31:51.182501+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":789929,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3775,"num_write":3,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-04-01T13:31:54.592607+0000","last_change":"2024-04-01T13:31:51.234370+0000","last_active":"2024-04-01T13:31:54.592607+0000","last_peered":"2024-04-01T13:31:54.592607+0000","last_clean":"2024-04-01T13:30:24.494919+0000","last_became_active":"2024-04-01T13:31:51.234370+0000","last_became_peered":"2024-04-01T13:31:51.234370+0000","last_unstale":"2024-04-01T13:31:54.592607+0000","last_undegraded":"2024-04-01T13:31:54.592607+0000","last_fullsized":"2024-04-01T13:31:51.190732+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"36'16","reported_seq":93,"reported_epoch":38,"state":"active+undersized+degraded","last_fresh":"2024-04-01T13:31:53.182673+0000","last_change":"2024-04-01T13:31:52.211125+0000","last_active":"2024-04-01T13:31:53.182673+0000","last_peered":"2024-04-01T13:31:53.182673+0000","last_clean":"2024-04-01T13:30:51.678933+0000","last_became_active":"2024-04-01T13:31:51.253680+0000","last_became_peered":"2024-04-01T13:31:51.253680+0000","last_unstale":"2024-04-01T13:31:53.182673+0000","last_undegraded":"2024-04-01T13:31:51.184462+0000","last_fullsized":"2024-04-01T13:31:51.180830+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8403401,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":10130,"num_write":21,"num_write_kb":5949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":615603,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":43,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-04-01T13:31:53.180698+0000","last_change":"2024-04-01T13:31:51.259259+0000","last_active":"2024-04-01T13:31:53.180698+0000","last_peered":"2024-04-01T13:31:53.180698+0000","last_clean":"2024-04-01T13:30:34.565817+0000","last_became_active":"2024-04-01T13:31:51.259259+0000","last_became_peered":"2024-04-01T13:31:51.259259+0000","last_unstale":"2024-04-01T13:31:53.180698+0000","last_undegraded":"2024-04-01T13:31:53.180698+0000","last_fullsized":"2024-04-01T13:31:51.182211+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"26'35","reported_seq":119,"reported_epoch":38,"state":"active+undersized+degraded","last_fresh":"2024-04-01T13:31:53.166723+0000","last_change":"2024-04-01T13:31:52.194421+0000","last_active":"2024-04-01T13:31:53.166723+0000","last_peered":"2024-04-01T13:31:53.166723+0000","last_clean":"2024-04-01T13:31:39.629935+0000","last_became_active":"2024-04-01T13:31:51.230853+0000","last_became_peered":"2024-04-01T13:31:51.230853+0000","last_unstale":"2024-04-01T13:31:53.166723+0000","last_undegraded":"2024-04-01T13:31:51.160053+0000","last_fullsized":"2024-04-01T13:31:51.158525+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":12168103,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":5339,"num_write":35,"num_write_kb":8781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":30,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-04-01T13:31:54.587030+0000","last_change":"2024-04-01T13:31:51.221067+0000","last_active":"2024-04-01T13:31:54.587030+0000","last_peered":"2024-04-01T13:31:54.587030+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:31:51.221067+0000","last_became_peered":"2024-04-01T13:31:51.221067+0000","last_unstale":"2024-04-01T13:31:54.587030+0000","last_undegraded":"2024-04-01T13:31:54.587030+0000","last_fullsized":"2024-04-01T13:31:51.181633+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"26'9","reported_seq":108,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-01T13:31:55.179260+0000","last_change":"2024-04-01T13:31:55.183397+0000","last_active":"2024-04-01T13:31:55.179260+0000","last_peered":"2024-04-01T13:31:55.179260+0000","last_clean":"2024-04-01T13:31:55.179260+0000","last_became_active":"2024-04-01T13:31:30.573263+0000","last_became_peered":"2024-04-01T13:31:30.573263+0000","last_unstale":"2024-04-01T13:31:55.179260+0000","last_undegraded":"2024-04-01T13:31:55.179260+0000","last_fullsized":"2024-04-01T13:31:55.179260+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"26'15","reported_seq":18,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-04-01T13:31:53.180140+0000","last_change":"2024-04-01T13:31:52.210237+0000","last_active":"2024-04-01T13:31:53.180140+0000","last_peered":"2024-04-01T13:31:53.180140+0000","last_clean":"2024-04-01T13:30:29.519276+0000","last_became_active":"2024-04-01T13:31:51.264104+0000","last_became_peered":"2024-04-01T13:31:51.264104+0000","last_unstale":"2024-04-01T13:31:53.180140+0000","last_undegraded":"2024-04-01T13:31:53.180140+0000","last_fullsized":"2024-04-01T13:31:51.198426+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-01T13:31:55.187527+0000","last_change":"2024-04-01T13:31:30.584389+0000","last_active":"2024-04-01T13:31:55.187527+0000","last_peered":"2024-04-01T13:31:55.187527+0000","last_clean":"2024-04-01T13:31:55.187527+0000","last_became_active":"2024-04-01T13:31:30.579930+0000","last_became_peered":"2024-04-01T13:31:30.579930+0000","last_unstale":"2024-04-01T13:31:55.187527+0000","last_undegraded":"2024-04-01T13:31:55.187527+0000","last_fullsized":"2024-04-01T13:31:55.187527+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"16'6","reported_seq":101,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-01T13:31:55.186432+0000","last_change":"2024-04-01T13:31:55.190835+0000","last_active":"2024-04-01T13:31:55.186432+0000","last_peered":"2024-04-01T13:31:55.186432+0000","last_clean":"2024-04-01T13:31:55.186432+0000","last_became_active":"2024-04-01T13:31:30.581362+0000","last_became_peered":"2024-04-01T13:31:30.581362+0000","last_unstale":"2024-04-01T13:31:55.186432+0000","last_undegraded":"2024-04-01T13:31:55.186432+0000","last_fullsized":"2024-04-01T13:31:55.186432+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4907578,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5282,"num_write":8,"num_write_kb":2191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-01T13:31:53.176537+0000","last_change":"2024-04-01T13:31:30.608391+0000","last_active":"2024-04-01T13:31:53.176537+0000","last_peered":"2024-04-01T13:31:53.176537+0000","last_clean":"2024-04-01T13:31:53.176537+0000","last_became_active":"2024-04-01T13:31:30.601821+0000","last_became_peered":"2024-04-01T13:31:30.601821+0000","last_unstale":"2024-04-01T13:31:53.176537+0000","last_undegraded":"2024-04-01T13:31:53.176537+0000","last_fullsized":"2024-04-01T13:31:53.176537+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"36'27","reported_seq":104,"reported_epoch":38,"state":"active+undersized+degraded","last_fresh":"2024-04-01T13:31:53.168270+0000","last_change":"2024-04-01T13:31:52.192592+0000","last_active":"2024-04-01T13:31:53.168270+0000","last_peered":"2024-04-01T13:31:53.168270+0000","last_clean":"2024-04-01T13:30:51.627795+0000","last_became_active":"2024-04-01T13:31:51.242351+0000","last_became_peered":"2024-04-01T13:31:51.242351+0000","last_unstale":"2024-04-01T13:31:53.168270+0000","last_undegraded":"2024-04-01T13:31:51.166005+0000","last_fullsized":"2024-04-01T13:31:51.163477+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739541,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":12139,"num_write":31,"num_write_kb":8248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":62874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-01T13:31:55.187707+0000","last_change":"2024-04-01T13:31:30.594953+0000","last_active":"2024-04-01T13:31:55.187707+0000","last_peered":"2024-04-01T13:31:55.187707+0000","last_clean":"2024-04-01T13:31:55.187707+0000","last_became_active":"2024-04-01T13:31:30.586687+0000","last_became_peered":"2024-04-01T13:31:30.586687+0000","last_unstale":"2024-04-01T13:31:55.187707+0000","last_undegraded":"2024-04-01T13:31:55.187707+0000","last_fullsized":"2024-04-01T13:31:55.187707+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":87,"reported_epoch":40,"state":"active+undersized","last_fresh":"2024-04-01T13:31:55.184081+0000","last_change":"2024-04-01T13:31:55.203175+0000","last_active":"2024-04-01T13:31:55.184081+0000","last_peered":"2024-04-01T13:31:55.184081+0000","last_clean":"2024-04-01T13:31:39.598475+0000","last_became_active":"2024-04-01T13:31:51.243464+0000","last_became_peered":"2024-04-01T13:31:51.243464+0000","last_unstale":"2024-04-01T13:31:55.184081+0000","last_undegraded":"2024-04-01T13:31:55.184081+0000","last_fullsized":"2024-04-01T13:31:51.176983+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-04-01T13:31:54.588550+0000","last_change":"2024-04-01T13:31:51.242909+0000","last_active":"2024-04-01T13:31:54.588550+0000","last_peered":"2024-04-01T13:31:54.588550+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:31:51.242909+0000","last_became_peered":"2024-04-01T13:31:51.242909+0000","last_unstale":"2024-04-01T13:31:54.588550+0000","last_undegraded":"2024-04-01T13:31:54.588550+0000","last_fullsized":"2024-04-01T13:31:51.185258+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":84874294,"num_objects":34,"num_object_clones":2,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":174,"num_read_kb":75723,"num_write":207,"num_write_kb":56435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":678477,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":261,"ondisk_log_size":261,"up":36,"acting":36,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76496,"kb_used_data":49480,"kb_used_omap":0,"kb_used_meta":27008,"kb_avail":94295344,"statfs":{"total":96636764160,"available":96558432256,"internally_reserved":0,"allocated":50667520,"data_stored":50598681,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27656192},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64156,"kb_used_data":37332,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94307684,"statfs":{"total":96636764160,"available":96571068416,"internally_reserved":0,"allocated":38227968,"data_stored":38248979,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69828,"kb_used_data":42448,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94302012,"statfs":{"total":96636764160,"available":96565260288,"internally_reserved":0,"allocated":43466752,"data_stored":44221816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":29,"seq":124554051609,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73084,"kb_used_data":46132,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94298756,"statfs":{"total":96636764160,"available":96561926144,"internally_reserved":0,"allocated":47239168,"data_stored":48357824,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1508,"internal_metadata":27589148},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-01T13:31:56.486 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-01T13:31:56.486 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-01T13:31:56.487 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [1, 2, 3, 0] 2024-04-01T13:31:56.487 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-01T13:31:56.487 INFO:teuthology.orchestra.run:waiting for 300 2024-04-01T13:31:56.545 INFO:tasks.ceph.osd.2:Stopped 2024-04-01T13:31:56.712 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T13:31:56.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:56.836+0000 7f0a6eb1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:56.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:31:56.842+0000 7f0a6eb1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:31:56.854 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-01T13:31:57.153 INFO:tasks.rados.rados.0.smithi059.stdout:125: done (7 left) 2024-04-01T13:31:57.153 INFO:tasks.rados.rados.0.smithi059.stdout:126: write oid 36 current snap is 6 2024-04-01T13:31:57.153 INFO:tasks.rados.rados.0.smithi059.stdout:116: finishing write tid 2 to smithi05931183-28 2024-04-01T13:31:57.153 INFO:tasks.rados.rados.0.smithi059.stdout:126: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-04-01T13:31:57.159 INFO:tasks.rados.rados.0.smithi059.stdout:126: writing smithi05931183-36 from 594981 to 1204178 tid 1 2024-04-01T13:31:57.168 INFO:tasks.rados.rados.0.smithi059.stdout:126: writing smithi05931183-36 from 1974123 to 2719151 tid 2 2024-04-01T13:31:57.172 INFO:tasks.rados.rados.0.smithi059.stdout:126: writing smithi05931183-36 from 3279873 to 3394981 tid 3 2024-04-01T13:31:57.172 INFO:tasks.rados.rados.0.smithi059.stdout:127: write (excl) oid 35 current snap is 6 2024-04-01T13:31:57.172 INFO:tasks.rados.rados.0.smithi059.stdout:118: finishing write tid 1 to smithi05931183-49 2024-04-01T13:31:57.172 INFO:tasks.rados.rados.0.smithi059.stdout:127: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-04-01T13:31:57.177 INFO:tasks.rados.rados.0.smithi059.stdout:127: writing smithi05931183-35 from 611788 to 1012587 tid 1 2024-04-01T13:31:57.185 INFO:tasks.rados.rados.0.smithi059.stdout:127: writing smithi05931183-35 from 1748958 to 2237644 tid 2 2024-04-01T13:31:57.193 INFO:tasks.rados.rados.0.smithi059.stdout:127: writing smithi05931183-35 from 2907296 to 3411788 tid 3 2024-04-01T13:31:57.193 INFO:tasks.rados.rados.0.smithi059.stdout:128: write oid 2 current snap is 6 2024-04-01T13:31:57.193 INFO:tasks.rados.rados.0.smithi059.stdout:128: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-04-01T13:31:57.201 INFO:tasks.rados.rados.0.smithi059.stdout:128: writing smithi05931183-2 from 628595 to 1104643 tid 1 2024-04-01T13:31:57.211 INFO:tasks.rados.rados.0.smithi059.stdout:128: writing smithi05931183-2 from 1807440 to 2439784 tid 2 2024-04-01T13:31:57.219 INFO:tasks.rados.rados.0.smithi059.stdout:128: writing smithi05931183-2 from 2934719 to 3387193 tid 3 2024-04-01T13:31:57.219 INFO:tasks.rados.rados.0.smithi059.stdout:129: read oid 9 snap -1 2024-04-01T13:31:57.220 INFO:tasks.rados.rados.0.smithi059.stdout:129: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-01T13:31:57.220 INFO:tasks.rados.rados.0.smithi059.stdout:130: read oid 46 snap -1 2024-04-01T13:31:57.220 INFO:tasks.rados.rados.0.smithi059.stdout:130: expect (ObjNum 62 snap 2 seq_num 62) 2024-04-01T13:31:57.220 INFO:tasks.rados.rados.0.smithi059.stdout:131: write oid 1 current snap is 6 2024-04-01T13:31:57.220 INFO:tasks.rados.rados.0.smithi059.stdout:117: finishing write tid 1 to smithi05931183-50 2024-04-01T13:31:57.220 INFO:tasks.rados.rados.0.smithi059.stdout:116: finishing write tid 3 to smithi05931183-28 2024-04-01T13:31:57.220 INFO:tasks.rados.rados.0.smithi059.stdout:118: finishing write tid 2 to smithi05931183-49 2024-04-01T13:31:57.220 INFO:tasks.rados.rados.0.smithi059.stdout:118: finishing write tid 3 to smithi05931183-49 2024-04-01T13:31:57.220 INFO:tasks.rados.rados.0.smithi059.stdout:117: finishing write tid 2 to smithi05931183-50 2024-04-01T13:31:57.220 INFO:tasks.rados.rados.0.smithi059.stdout:131: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2024-04-01T13:31:57.228 INFO:tasks.rados.rados.0.smithi059.stdout:131: writing smithi05931183-1 from 645402 to 1196699 tid 1 2024-04-01T13:31:57.234 INFO:tasks.rados.rados.0.smithi059.stdout:131: writing smithi05931183-1 from 1749569 to 2241924 tid 2 2024-04-01T13:31:57.241 INFO:tasks.rados.rados.0.smithi059.stdout:131: writing smithi05931183-1 from 2845789 to 3445402 tid 3 2024-04-01T13:31:57.241 INFO:tasks.rados.rados.0.smithi059.stdout:132: read oid 19 snap -1 2024-04-01T13:31:57.241 INFO:tasks.rados.rados.0.smithi059.stdout:132: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-01T13:31:57.242 INFO:tasks.rados.rados.0.smithi059.stdout:133: write oid 43 current snap is 6 2024-04-01T13:31:57.242 INFO:tasks.rados.rados.0.smithi059.stdout:116: finishing write tid 4 to smithi05931183-28 2024-04-01T13:31:57.242 INFO:tasks.rados.rados.0.smithi059.stdout:118: finishing write tid 4 to smithi05931183-49 2024-04-01T13:31:57.242 INFO:tasks.rados.rados.0.smithi059.stdout:116: finishing write tid 5 to smithi05931183-28 2024-04-01T13:31:57.242 INFO:tasks.rados.rados.0.smithi059.stdout:116: oid 28 updating version 0 to 20 2024-04-01T13:31:57.242 INFO:tasks.rados.rados.0.smithi059.stdout:116: oid 28 updating version 20 to 21 2024-04-01T13:31:57.242 INFO:tasks.rados.rados.0.smithi059.stdout:116: oid 28 version 21 is already newer than 18 2024-04-01T13:31:57.242 INFO:tasks.rados.rados.0.smithi059.stdout:116: oid 28 version 21 is already newer than 19 2024-04-01T13:31:57.242 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 28 v 21 (ObjNum 78 snap 6 seq_num 78) dirty exists 2024-04-01T13:31:57.242 INFO:tasks.rados.rados.0.smithi059.stdout:116: left oid 28 (ObjNum 78 snap 6 seq_num 78) 2024-04-01T13:31:57.242 INFO:tasks.rados.rados.0.smithi059.stdout:118: finishing write tid 5 to smithi05931183-49 2024-04-01T13:31:57.242 INFO:tasks.rados.rados.0.smithi059.stdout:118: oid 49 updating version 0 to 38 2024-04-01T13:31:57.242 INFO:tasks.rados.rados.0.smithi059.stdout:118: oid 49 version 38 is already newer than 36 2024-04-01T13:31:57.242 INFO:tasks.rados.rados.0.smithi059.stdout:118: oid 49 version 38 is already newer than 38 2024-04-01T13:31:57.243 INFO:tasks.rados.rados.0.smithi059.stdout:118: oid 49 updating version 38 to 39 2024-04-01T13:31:57.243 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 49 v 39 (ObjNum 80 snap 6 seq_num 80) dirty exists 2024-04-01T13:31:57.243 INFO:tasks.rados.rados.0.smithi059.stdout:118: left oid 49 (ObjNum 80 snap 6 seq_num 80) 2024-04-01T13:31:57.243 INFO:tasks.rados.rados.0.smithi059.stdout:119: finishing write tid 1 to smithi05931183-26 2024-04-01T13:31:57.243 INFO:tasks.rados.rados.0.smithi059.stdout:133: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2024-04-01T13:31:57.248 INFO:tasks.rados.rados.0.smithi059.stdout:133: writing smithi05931183-43 from 662209 to 1288755 tid 1 2024-04-01T13:31:57.254 INFO:tasks.rados.rados.0.smithi059.stdout:133: writing smithi05931183-43 from 1808051 to 2444064 tid 2 2024-04-01T13:31:57.260 INFO:tasks.rados.rados.0.smithi059.stdout:133: writing smithi05931183-43 from 2873212 to 3462209 tid 3 2024-04-01T13:31:57.260 INFO:tasks.rados.rados.0.smithi059.stdout:116: done (14 left) 2024-04-01T13:31:57.261 INFO:tasks.rados.rados.0.smithi059.stdout:118: done (13 left) 2024-04-01T13:31:57.261 INFO:tasks.rados.rados.0.smithi059.stdout:134: snap_create 2024-04-01T13:31:57.261 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 13 2024-04-01T13:31:57.261 INFO:tasks.rados.rados.0.smithi059.stdout:121: expect (ObjNum 68 snap 3 seq_num 68) 2024-04-01T13:31:57.424 INFO:tasks.rados.rados.0.smithi059.stdout:117: finishing write tid 3 to smithi05931183-50 2024-04-01T13:31:57.424 INFO:tasks.rados.rados.0.smithi059.stdout:120: finishing write tid 1 to smithi05931183-44 2024-04-01T13:31:57.424 INFO:tasks.rados.rados.0.smithi059.stdout:119: finishing write tid 2 to smithi05931183-26 2024-04-01T13:31:57.425 INFO:tasks.rados.rados.0.smithi059.stdout:117: finishing write tid 4 to smithi05931183-50 2024-04-01T13:31:57.425 INFO:tasks.rados.rados.0.smithi059.stdout:119: finishing write tid 3 to smithi05931183-26 2024-04-01T13:31:57.425 INFO:tasks.rados.rados.0.smithi059.stdout:117: finishing write tid 5 to smithi05931183-50 2024-04-01T13:31:57.425 INFO:tasks.rados.rados.0.smithi059.stdout:117: oid 50 updating version 0 to 17 2024-04-01T13:31:57.425 INFO:tasks.rados.rados.0.smithi059.stdout:117: oid 50 version 17 is already newer than 16 2024-04-01T13:31:57.425 INFO:tasks.rados.rados.0.smithi059.stdout:117: oid 50 version 17 is already newer than 14 2024-04-01T13:31:57.425 INFO:tasks.rados.rados.0.smithi059.stdout:117: oid 50 version 17 is already newer than 15 2024-04-01T13:31:57.425 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 50 v 17 (ObjNum 79 snap 6 seq_num 79) dirty exists 2024-04-01T13:31:57.425 INFO:tasks.rados.rados.0.smithi059.stdout:117: left oid 50 (ObjNum 79 snap 6 seq_num 79) 2024-04-01T13:31:57.425 INFO:tasks.rados.rados.0.smithi059.stdout:120: finishing write tid 2 to smithi05931183-44 2024-04-01T13:31:57.425 INFO:tasks.rados.rados.0.smithi059.stdout:119: finishing write tid 4 to smithi05931183-26 2024-04-01T13:31:57.425 INFO:tasks.rados.rados.0.smithi059.stdout:119: finishing write tid 5 to smithi05931183-26 2024-04-01T13:31:57.425 INFO:tasks.rados.rados.0.smithi059.stdout:119: oid 26 updating version 0 to 26 2024-04-01T13:31:57.425 INFO:tasks.rados.rados.0.smithi059.stdout:119: oid 26 updating version 26 to 29 2024-04-01T13:31:57.426 INFO:tasks.rados.rados.0.smithi059.stdout:119: oid 26 version 29 is already newer than 27 2024-04-01T13:31:57.426 INFO:tasks.rados.rados.0.smithi059.stdout:119: oid 26 version 29 is already newer than 29 2024-04-01T13:31:57.426 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 26 v 29 (ObjNum 81 snap 6 seq_num 81) dirty exists 2024-04-01T13:31:57.426 INFO:tasks.rados.rados.0.smithi059.stdout:119: left oid 26 (ObjNum 81 snap 6 seq_num 81) 2024-04-01T13:31:57.426 INFO:tasks.rados.rados.0.smithi059.stdout:120: finishing write tid 3 to smithi05931183-44 2024-04-01T13:31:57.426 INFO:tasks.rados.rados.0.smithi059.stdout:117: done (12 left) 2024-04-01T13:31:57.426 INFO:tasks.rados.rados.0.smithi059.stdout:119: done (11 left) 2024-04-01T13:31:57.426 INFO:tasks.rados.rados.0.smithi059.stdout:121: done (10 left) 2024-04-01T13:31:57.426 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 10 2024-04-01T13:31:57.426 INFO:tasks.rados.rados.0.smithi059.stdout:120: finishing write tid 4 to smithi05931183-44 2024-04-01T13:31:57.426 INFO:tasks.rados.rados.0.smithi059.stdout:120: finishing write tid 5 to smithi05931183-44 2024-04-01T13:31:57.426 INFO:tasks.rados.rados.0.smithi059.stdout:120: oid 44 updating version 0 to 43 2024-04-01T13:31:57.426 INFO:tasks.rados.rados.0.smithi059.stdout:120: oid 44 updating version 43 to 44 2024-04-01T13:31:57.426 INFO:tasks.rados.rados.0.smithi059.stdout:120: oid 44 version 44 is already newer than 41 2024-04-01T13:31:57.427 INFO:tasks.rados.rados.0.smithi059.stdout:120: oid 44 version 44 is already newer than 42 2024-04-01T13:31:57.427 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 44 v 44 (ObjNum 82 snap 6 seq_num 82) dirty exists 2024-04-01T13:31:57.427 INFO:tasks.rados.rados.0.smithi059.stdout:120: left oid 44 (ObjNum 82 snap 6 seq_num 82) 2024-04-01T13:31:57.427 INFO:tasks.rados.rados.0.smithi059.stdout:123: expect (ObjNum 36 snap 0 seq_num 36) 2024-04-01T13:31:57.523 INFO:tasks.rados.rados.0.smithi059.stdout:126: finishing write tid 1 to smithi05931183-36 2024-04-01T13:31:57.523 INFO:tasks.rados.rados.0.smithi059.stdout:129: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-01T13:31:57.606 INFO:tasks.rados.rados.0.smithi059.stdout:133: finishing write tid 1 to smithi05931183-43 2024-04-01T13:31:57.606 INFO:tasks.rados.rados.0.smithi059.stdout:132: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-01T13:31:57.695 INFO:tasks.rados.rados.0.smithi059.stdout:126: finishing write tid 2 to smithi05931183-36 2024-04-01T13:31:57.695 INFO:tasks.rados.rados.0.smithi059.stdout:133: finishing write tid 2 to smithi05931183-43 2024-04-01T13:31:57.695 INFO:tasks.rados.rados.0.smithi059.stdout:126: finishing write tid 3 to smithi05931183-36 2024-04-01T13:31:57.695 INFO:tasks.rados.rados.0.smithi059.stdout:126: finishing write tid 4 to smithi05931183-36 2024-04-01T13:31:57.695 INFO:tasks.rados.rados.0.smithi059.stdout:133: finishing write tid 3 to smithi05931183-43 2024-04-01T13:31:57.695 INFO:tasks.rados.rados.0.smithi059.stdout:126: finishing write tid 5 to smithi05931183-36 2024-04-01T13:31:57.695 INFO:tasks.rados.rados.0.smithi059.stdout:126: oid 36 updating version 0 to 10 2024-04-01T13:31:57.695 INFO:tasks.rados.rados.0.smithi059.stdout:126: oid 36 updating version 10 to 11 2024-04-01T13:31:57.695 INFO:tasks.rados.rados.0.smithi059.stdout:126: oid 36 updating version 11 to 13 2024-04-01T13:31:57.695 INFO:tasks.rados.rados.0.smithi059.stdout:126: oid 36 version 13 is already newer than 13 2024-04-01T13:31:57.695 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 36 v 13 (ObjNum 83 snap 6 seq_num 83) dirty exists 2024-04-01T13:31:57.695 INFO:tasks.rados.rados.0.smithi059.stdout:126: left oid 36 (ObjNum 83 snap 6 seq_num 83) 2024-04-01T13:31:57.696 INFO:tasks.rados.rados.0.smithi059.stdout:133: finishing write tid 4 to smithi05931183-43 2024-04-01T13:31:57.696 INFO:tasks.rados.rados.0.smithi059.stdout:120: done (9 left) 2024-04-01T13:31:57.696 INFO:tasks.rados.rados.0.smithi059.stdout:123: done (8 left) 2024-04-01T13:31:57.696 INFO:tasks.rados.rados.0.smithi059.stdout:126: done (7 left) 2024-04-01T13:31:57.696 INFO:tasks.rados.rados.0.smithi059.stdout:129: done (6 left) 2024-04-01T13:31:57.696 INFO:tasks.rados.rados.0.smithi059.stdout:132: done (5 left) 2024-04-01T13:31:57.696 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 5 2024-04-01T13:31:57.696 INFO:tasks.rados.rados.0.smithi059.stdout:133: finishing write tid 5 to smithi05931183-43 2024-04-01T13:31:57.696 INFO:tasks.rados.rados.0.smithi059.stdout:133: oid 43 updating version 0 to 28 2024-04-01T13:31:57.696 INFO:tasks.rados.rados.0.smithi059.stdout:133: oid 43 version 28 is already newer than 26 2024-04-01T13:31:57.696 INFO:tasks.rados.rados.0.smithi059.stdout:133: oid 43 updating version 28 to 29 2024-04-01T13:31:57.696 INFO:tasks.rados.rados.0.smithi059.stdout:133: oid 43 version 29 is already newer than 28 2024-04-01T13:31:57.696 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 43 v 29 (ObjNum 87 snap 6 seq_num 87) dirty exists 2024-04-01T13:31:57.696 INFO:tasks.rados.rados.0.smithi059.stdout:133: left oid 43 (ObjNum 87 snap 6 seq_num 87) 2024-04-01T13:31:57.697 INFO:tasks.rados.rados.0.smithi059.stdout:133: done (4 left) 2024-04-01T13:31:57.697 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 4 2024-04-01T13:31:58.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~33s 2024-04-01T13:32:01.547 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [1, 3, 0] 2024-04-01T13:32:01.547 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-01T13:32:01.547 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T13:32:01.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:32:01.685+0000 7fbcba71d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:32:01.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:32:01.693+0000 7fbcba71d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:32:01.875 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-01T13:32:01.876 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-01T13:32:01.888 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-04-01T13:32:01.388708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92506762,"num_objects":36,"num_object_clones":1,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":317,"num_read_kb":100711,"num_write":382,"num_write_kb":70195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":678477,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":289188,"kb_used_data":177284,"kb_used_omap":1,"kb_used_meta":107582,"kb_avail":377198172,"statfs":{"total":386547056640,"available":386250928128,"internally_reserved":0,"allocated":181538816,"data_stored":182102762,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":1508,"internal_metadata":110164508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7042100,"num_objects":0,"num_object_clones":-1,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":40,"num_read_kb":24897,"num_write":38,"num_write_kb":11866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.227093"},"pg_stats":[{"pgid":"3.f","version":"42'13","reported_seq":82,"reported_epoch":42,"state":"active+undersized+degraded","last_fresh":"2024-04-01T13:31:57.363861+0000","last_change":"2024-04-01T13:31:57.223588+0000","last_active":"2024-04-01T13:31:57.363861+0000","last_peered":"2024-04-01T13:31:57.363861+0000","last_clean":"2024-04-01T13:30:45.695648+0000","last_became_active":"2024-04-01T13:31:51.253156+0000","last_became_peered":"2024-04-01T13:31:51.253156+0000","last_unstale":"2024-04-01T13:31:57.363861+0000","last_undegraded":"2024-04-01T13:31:51.171139+0000","last_fullsized":"2024-04-01T13:31:51.169385+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":9787,"num_write":17,"num_write_kb":4485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"26'11","reported_seq":97,"reported_epoch":40,"state":"active+undersized+degraded","last_fresh":"2024-04-01T13:31:55.193722+0000","last_change":"2024-04-01T13:31:55.204853+0000","last_active":"2024-04-01T13:31:55.193722+0000","last_peered":"2024-04-01T13:31:55.193722+0000","last_clean":"2024-04-01T13:31:39.599913+0000","last_became_active":"2024-04-01T13:31:51.231828+0000","last_became_peered":"2024-04-01T13:31:51.231828+0000","last_unstale":"2024-04-01T13:31:55.193722+0000","last_undegraded":"2024-04-01T13:31:51.173803+0000","last_fullsized":"2024-04-01T13:31:51.171147+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042034,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"40'31","reported_seq":86,"reported_epoch":42,"state":"active+undersized+degraded+snaptrim","last_fresh":"2024-04-01T13:31:57.219801+0000","last_change":"2024-04-01T13:31:57.218783+0000","last_active":"2024-04-01T13:31:57.219801+0000","last_peered":"2024-04-01T13:31:57.219801+0000","last_clean":"2024-04-01T13:30:34.592347+0000","last_became_active":"2024-04-01T13:31:51.268999+0000","last_became_peered":"2024-04-01T13:31:51.268999+0000","last_unstale":"2024-04-01T13:31:57.219801+0000","last_undegraded":"2024-04-01T13:31:51.189944+0000","last_fullsized":"2024-04-01T13:31:51.186639+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":10823576,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":28,"num_write_kb":7879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"37'28","reported_seq":109,"reported_epoch":42,"state":"active+clean+snaptrim","last_fresh":"2024-04-01T13:31:57.228400+0000","last_change":"2024-04-01T13:31:57.227765+0000","last_active":"2024-04-01T13:31:57.228400+0000","last_peered":"2024-04-01T13:31:57.228400+0000","last_clean":"2024-04-01T13:31:57.228400+0000","last_became_active":"2024-04-01T13:31:30.574937+0000","last_became_peered":"2024-04-01T13:31:30.574937+0000","last_unstale":"2024-04-01T13:31:57.228400+0000","last_undegraded":"2024-04-01T13:31:57.228400+0000","last_fullsized":"2024-04-01T13:31:57.228400+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"40'19","reported_seq":122,"reported_epoch":42,"state":"active+clean+snaptrim","last_fresh":"2024-04-01T13:31:57.224455+0000","last_change":"2024-04-01T13:31:57.224070+0000","last_active":"2024-04-01T13:31:57.224455+0000","last_peered":"2024-04-01T13:31:57.224455+0000","last_clean":"2024-04-01T13:31:57.224455+0000","last_became_active":"2024-04-01T13:31:30.604705+0000","last_became_peered":"2024-04-01T13:31:30.604705+0000","last_unstale":"2024-04-01T13:31:57.224455+0000","last_undegraded":"2024-04-01T13:31:57.224455+0000","last_fullsized":"2024-04-01T13:31:57.224455+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":10319366,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":17307,"num_write":21,"num_write_kb":6206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"37'15","reported_seq":94,"reported_epoch":42,"state":"active+undersized+degraded","last_fresh":"2024-04-01T13:31:57.228592+0000","last_change":"2024-04-01T13:31:57.232023+0000","last_active":"2024-04-01T13:31:57.228592+0000","last_peered":"2024-04-01T13:31:57.228592+0000","last_clean":"2024-04-01T13:30:49.664233+0000","last_became_active":"2024-04-01T13:31:51.233044+0000","last_became_peered":"2024-04-01T13:31:51.233044+0000","last_unstale":"2024-04-01T13:31:57.228592+0000","last_undegraded":"2024-04-01T13:31:51.180750+0000","last_fullsized":"2024-04-01T13:31:51.179132+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":8521050,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":18414,"num_write":16,"num_write_kb":4632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"37'25","reported_seq":93,"reported_epoch":42,"state":"active+undersized+degraded","last_fresh":"2024-04-01T13:31:57.222937+0000","last_change":"2024-04-01T13:31:57.222937+0000","last_active":"2024-04-01T13:31:57.222937+0000","last_peered":"2024-04-01T13:31:57.222937+0000","last_clean":"2024-04-01T13:30:49.030107+0000","last_became_active":"2024-04-01T13:31:51.222200+0000","last_became_peered":"2024-04-01T13:31:51.222200+0000","last_unstale":"2024-04-01T13:31:57.222937+0000","last_undegraded":"2024-04-01T13:31:51.166334+0000","last_fullsized":"2024-04-01T13:31:51.164616+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":6016840,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":27,"num_write_kb":7776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"24'20","reported_seq":35,"reported_epoch":42,"state":"active+undersized","last_fresh":"2024-04-01T13:31:57.220862+0000","last_change":"2024-04-01T13:31:57.220862+0000","last_active":"2024-04-01T13:31:57.220862+0000","last_peered":"2024-04-01T13:31:57.220862+0000","last_clean":"2024-04-01T13:30:29.519276+0000","last_became_active":"2024-04-01T13:31:51.243345+0000","last_became_peered":"2024-04-01T13:31:51.243345+0000","last_unstale":"2024-04-01T13:31:57.220862+0000","last_undegraded":"2024-04-01T13:31:57.220862+0000","last_fullsized":"2024-04-01T13:31:51.177096+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":4962,"num_write":1,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":211,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-01T13:31:57.198062+0000","last_change":"2024-04-01T13:31:30.585158+0000","last_active":"2024-04-01T13:31:57.198062+0000","last_peered":"2024-04-01T13:31:57.198062+0000","last_clean":"2024-04-01T13:31:57.198062+0000","last_became_active":"2024-04-01T13:31:30.578315+0000","last_became_peered":"2024-04-01T13:31:30.578315+0000","last_unstale":"2024-04-01T13:31:57.198062+0000","last_undegraded":"2024-04-01T13:31:57.198062+0000","last_fullsized":"2024-04-01T13:31:57.198062+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:22.378617+0000","last_clean_scrub_stamp":"2024-04-01T13:30:22.378617+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":42,"state":"active+undersized","last_fresh":"2024-04-01T13:31:57.208588+0000","last_change":"2024-04-01T13:31:51.226808+0000","last_active":"2024-04-01T13:31:57.208588+0000","last_peered":"2024-04-01T13:31:57.208588+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:31:51.226808+0000","last_became_peered":"2024-04-01T13:31:51.226808+0000","last_unstale":"2024-04-01T13:31:57.208588+0000","last_undegraded":"2024-04-01T13:31:57.208588+0000","last_fullsized":"2024-04-01T13:31:51.158519+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"37'10","reported_seq":41,"reported_epoch":42,"state":"active+undersized","last_fresh":"2024-04-01T13:31:57.226863+0000","last_change":"2024-04-01T13:31:57.230620+0000","last_active":"2024-04-01T13:31:57.226863+0000","last_peered":"2024-04-01T13:31:57.226863+0000","last_clean":"2024-04-01T13:30:29.519276+0000","last_became_active":"2024-04-01T13:31:51.273685+0000","last_became_peered":"2024-04-01T13:31:51.273685+0000","last_unstale":"2024-04-01T13:31:57.226863+0000","last_undegraded":"2024-04-01T13:31:57.226863+0000","last_fullsized":"2024-04-01T13:31:51.182501+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":789929,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3775,"num_write":3,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":36,"reported_epoch":42,"state":"active+undersized","last_fresh":"2024-04-01T13:31:57.225024+0000","last_change":"2024-04-01T13:31:51.234370+0000","last_active":"2024-04-01T13:31:57.225024+0000","last_peered":"2024-04-01T13:31:57.225024+0000","last_clean":"2024-04-01T13:30:24.494919+0000","last_became_active":"2024-04-01T13:31:51.234370+0000","last_became_peered":"2024-04-01T13:31:51.234370+0000","last_unstale":"2024-04-01T13:31:57.225024+0000","last_undegraded":"2024-04-01T13:31:57.225024+0000","last_fullsized":"2024-04-01T13:31:51.190732+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"42'29","reported_seq":116,"reported_epoch":42,"state":"active+undersized+degraded+snaptrim","last_fresh":"2024-04-01T13:31:57.386959+0000","last_change":"2024-04-01T13:31:57.207790+0000","last_active":"2024-04-01T13:31:57.386959+0000","last_peered":"2024-04-01T13:31:57.386959+0000","last_clean":"2024-04-01T13:30:51.678933+0000","last_became_active":"2024-04-01T13:31:51.253680+0000","last_became_peered":"2024-04-01T13:31:51.253680+0000","last_unstale":"2024-04-01T13:31:57.386959+0000","last_undegraded":"2024-04-01T13:31:51.184462+0000","last_fullsized":"2024-04-01T13:31:51.180830+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":10016873,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":10132,"num_write":31,"num_write_kb":9292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":615603,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":53,"reported_epoch":42,"state":"active+undersized","last_fresh":"2024-04-01T13:31:57.205829+0000","last_change":"2024-04-01T13:31:51.259259+0000","last_active":"2024-04-01T13:31:57.205829+0000","last_peered":"2024-04-01T13:31:57.205829+0000","last_clean":"2024-04-01T13:30:34.565817+0000","last_became_active":"2024-04-01T13:31:51.259259+0000","last_became_peered":"2024-04-01T13:31:51.259259+0000","last_unstale":"2024-04-01T13:31:57.205829+0000","last_undegraded":"2024-04-01T13:31:57.205829+0000","last_fullsized":"2024-04-01T13:31:51.182211+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"42'48","reported_seq":148,"reported_epoch":42,"state":"active+undersized+degraded+snaptrim","last_fresh":"2024-04-01T13:31:57.318332+0000","last_change":"2024-04-01T13:31:57.193289+0000","last_active":"2024-04-01T13:31:57.318332+0000","last_peered":"2024-04-01T13:31:57.318332+0000","last_clean":"2024-04-01T13:31:39.629935+0000","last_became_active":"2024-04-01T13:31:51.230853+0000","last_became_peered":"2024-04-01T13:31:51.230853+0000","last_unstale":"2024-04-01T13:31:57.318332+0000","last_undegraded":"2024-04-01T13:31:51.160053+0000","last_fullsized":"2024-04-01T13:31:51.158525+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":15831996,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":14239,"num_write":45,"num_write_kb":11974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":37,"reported_epoch":42,"state":"active+undersized","last_fresh":"2024-04-01T13:31:57.233066+0000","last_change":"2024-04-01T13:31:51.221067+0000","last_active":"2024-04-01T13:31:57.233066+0000","last_peered":"2024-04-01T13:31:57.233066+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:31:51.221067+0000","last_became_peered":"2024-04-01T13:31:51.221067+0000","last_unstale":"2024-04-01T13:31:57.233066+0000","last_undegraded":"2024-04-01T13:31:57.233066+0000","last_fullsized":"2024-04-01T13:31:51.181633+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"26'9","reported_seq":108,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-01T13:31:55.179260+0000","last_change":"2024-04-01T13:31:55.183397+0000","last_active":"2024-04-01T13:31:55.179260+0000","last_peered":"2024-04-01T13:31:55.179260+0000","last_clean":"2024-04-01T13:31:55.179260+0000","last_became_active":"2024-04-01T13:31:30.573263+0000","last_became_peered":"2024-04-01T13:31:30.573263+0000","last_unstale":"2024-04-01T13:31:55.179260+0000","last_undegraded":"2024-04-01T13:31:55.179260+0000","last_fullsized":"2024-04-01T13:31:55.179260+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"26'15","reported_seq":31,"reported_epoch":42,"state":"active+undersized","last_fresh":"2024-04-01T13:31:57.233709+0000","last_change":"2024-04-01T13:31:57.271029+0000","last_active":"2024-04-01T13:31:57.233709+0000","last_peered":"2024-04-01T13:31:57.233709+0000","last_clean":"2024-04-01T13:30:29.519276+0000","last_became_active":"2024-04-01T13:31:51.264104+0000","last_became_peered":"2024-04-01T13:31:51.264104+0000","last_unstale":"2024-04-01T13:31:57.233709+0000","last_undegraded":"2024-04-01T13:31:57.233709+0000","last_fullsized":"2024-04-01T13:31:51.198426+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-01T13:31:55.187527+0000","last_change":"2024-04-01T13:31:30.584389+0000","last_active":"2024-04-01T13:31:55.187527+0000","last_peered":"2024-04-01T13:31:55.187527+0000","last_clean":"2024-04-01T13:31:55.187527+0000","last_became_active":"2024-04-01T13:31:30.579930+0000","last_became_peered":"2024-04-01T13:31:30.579930+0000","last_unstale":"2024-04-01T13:31:55.187527+0000","last_undegraded":"2024-04-01T13:31:55.187527+0000","last_fullsized":"2024-04-01T13:31:55.187527+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"16'6","reported_seq":101,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-01T13:31:55.186432+0000","last_change":"2024-04-01T13:31:55.190835+0000","last_active":"2024-04-01T13:31:55.186432+0000","last_peered":"2024-04-01T13:31:55.186432+0000","last_clean":"2024-04-01T13:31:55.186432+0000","last_became_active":"2024-04-01T13:31:30.581362+0000","last_became_peered":"2024-04-01T13:31:30.581362+0000","last_unstale":"2024-04-01T13:31:55.186432+0000","last_undegraded":"2024-04-01T13:31:55.186432+0000","last_fullsized":"2024-04-01T13:31:55.186432+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4907578,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5282,"num_write":8,"num_write_kb":2191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":81,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-01T13:31:57.216141+0000","last_change":"2024-04-01T13:31:30.608391+0000","last_active":"2024-04-01T13:31:57.216141+0000","last_peered":"2024-04-01T13:31:57.216141+0000","last_clean":"2024-04-01T13:31:57.216141+0000","last_became_active":"2024-04-01T13:31:30.601821+0000","last_became_peered":"2024-04-01T13:31:30.601821+0000","last_unstale":"2024-04-01T13:31:57.216141+0000","last_undegraded":"2024-04-01T13:31:57.216141+0000","last_fullsized":"2024-04-01T13:31:57.216141+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"36'27","reported_seq":120,"reported_epoch":42,"state":"active+undersized+degraded","last_fresh":"2024-04-01T13:31:57.337742+0000","last_change":"2024-04-01T13:31:57.221735+0000","last_active":"2024-04-01T13:31:57.337742+0000","last_peered":"2024-04-01T13:31:57.337742+0000","last_clean":"2024-04-01T13:30:51.627795+0000","last_became_active":"2024-04-01T13:31:51.242351+0000","last_became_peered":"2024-04-01T13:31:51.242351+0000","last_unstale":"2024-04-01T13:31:57.337742+0000","last_undegraded":"2024-04-01T13:31:51.166005+0000","last_fullsized":"2024-04-01T13:31:51.163477+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739541,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":16706,"num_write":31,"num_write_kb":8248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":62874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-01T13:31:55.187707+0000","last_change":"2024-04-01T13:31:30.594953+0000","last_active":"2024-04-01T13:31:55.187707+0000","last_peered":"2024-04-01T13:31:55.187707+0000","last_clean":"2024-04-01T13:31:55.187707+0000","last_became_active":"2024-04-01T13:31:30.586687+0000","last_became_peered":"2024-04-01T13:31:30.586687+0000","last_unstale":"2024-04-01T13:31:55.187707+0000","last_undegraded":"2024-04-01T13:31:55.187707+0000","last_fullsized":"2024-04-01T13:31:55.187707+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":87,"reported_epoch":40,"state":"active+undersized","last_fresh":"2024-04-01T13:31:55.184081+0000","last_change":"2024-04-01T13:31:55.203175+0000","last_active":"2024-04-01T13:31:55.184081+0000","last_peered":"2024-04-01T13:31:55.184081+0000","last_clean":"2024-04-01T13:31:39.598475+0000","last_became_active":"2024-04-01T13:31:51.243464+0000","last_became_peered":"2024-04-01T13:31:51.243464+0000","last_unstale":"2024-04-01T13:31:55.184081+0000","last_undegraded":"2024-04-01T13:31:55.184081+0000","last_fullsized":"2024-04-01T13:31:51.176983+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:29.519276+0000","last_clean_scrub_stamp":"2024-04-01T13:30:29.519276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":42,"state":"active+undersized","last_fresh":"2024-04-01T13:31:57.216361+0000","last_change":"2024-04-01T13:31:51.242909+0000","last_active":"2024-04-01T13:31:57.216361+0000","last_peered":"2024-04-01T13:31:57.216361+0000","last_clean":"2024-04-01T13:30:23.382260+0000","last_became_active":"2024-04-01T13:31:51.242909+0000","last_became_peered":"2024-04-01T13:31:51.242909+0000","last_unstale":"2024-04-01T13:31:57.216361+0000","last_undegraded":"2024-04-01T13:31:57.216361+0000","last_fullsized":"2024-04-01T13:31:51.185258+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T13:30:23.382260+0000","last_clean_scrub_stamp":"2024-04-01T13:30:23.382260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":91916394,"num_objects":34,"num_object_clones":1,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":214,"num_read_kb":100620,"num_write":245,"num_write_kb":68301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":678477,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":306,"ondisk_log_size":306,"up":36,"acting":36,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81508,"kb_used_data":52564,"kb_used_omap":0,"kb_used_meta":27008,"kb_avail":94290332,"statfs":{"total":96636764160,"available":96553299968,"internally_reserved":0,"allocated":53825536,"data_stored":53687972,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27656192},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64156,"kb_used_data":37332,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94307684,"statfs":{"total":96636764160,"available":96571068416,"internally_reserved":0,"allocated":38227968,"data_stored":38248979,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69828,"kb_used_data":42448,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94302012,"statfs":{"total":96636764160,"available":96565260288,"internally_reserved":0,"allocated":43466752,"data_stored":44221816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":29,"seq":124554051614,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73696,"kb_used_data":44940,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94298144,"statfs":{"total":96636764160,"available":96561299456,"internally_reserved":0,"allocated":46018560,"data_stored":45943995,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1508,"internal_metadata":27589148},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-01T13:32:01.889 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-backfill 3.0 3.7 2024-04-01T13:32:02.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:32:02.012+0000 7f3c0f71d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:32:02.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:32:02.022+0000 7f3c0f71d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:32:02.202 INFO:teuthology.orchestra.run.smithi059.stderr:instructing pg(s) [3.0,3.7] on osd.3 to force-backfill; 2024-04-01T13:32:02.202 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-04-01T13:32:04.079 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~38s 2024-04-01T13:32:07.215 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [1, 3, 0] 2024-04-01T13:32:07.215 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-01T13:32:07.215 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [1, 3, 0] 2024-04-01T13:32:07.216 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-01T13:32:07.216 INFO:teuthology.orchestra.run:waiting for 300 2024-04-01T13:32:07.293 INFO:tasks.ceph.osd.3:Stopped 2024-04-01T13:32:09.382 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~44s 2024-04-01T13:32:12.294 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2, 3] live_osds: [1, 0] 2024-04-01T13:32:12.294 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-01T13:32:12.294 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-04-01T13:32:12.294 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-01T13:32:12.295 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-01T13:32:12.298 INFO:tasks.ceph.osd.2:Started 2024-04-01T13:32:12.298 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-01T13:32:12.410 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:32:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:32:12.414 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-01T13:32:14.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~49s 2024-04-01T13:32:16.092 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-01T13:32:17.414 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-01T13:32:17.519 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-01T13:32:17.519 INFO:teuthology.orchestra.run.smithi087.stdout: "ops_in_flight": [ 2024-04-01T13:32:17.519 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:32:17.519 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "client_request", 2024-04-01T13:32:17.519 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 43, 2024-04-01T13:32:17.519 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:32:17.520 INFO:teuthology.orchestra.run.smithi087.stdout: "time_event": { 2024-04-01T13:32:17.520 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:32:17.520 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:32:17.156106+0000" 2024-04-01T13:32:17.520 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.520 INFO:teuthology.orchestra.run.smithi087.stdout: "blocking_event": { 2024-04-01T13:32:17.520 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:32:17.520 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:32:17.156149+0000" 2024-04-01T13:32:17.520 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.520 INFO:teuthology.orchestra.run.smithi087.stdout: "blocking_event": { 2024-04-01T13:32:17.520 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:32:17.520 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:32:17.157004+0000" 2024-04-01T13:32:17.520 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.520 INFO:teuthology.orchestra.run.smithi087.stdout: "blocking_event": { 2024-04-01T13:32:17.520 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:32:17.520 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:32:17.159303+0000" 2024-04-01T13:32:17.521 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.521 INFO:teuthology.orchestra.run.smithi087.stdout: "blocking_event": { 2024-04-01T13:32:17.521 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:32:17.521 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:32:17.181606+0000" 2024-04-01T13:32:17.521 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.521 INFO:teuthology.orchestra.run.smithi087.stdout: "blocking_event": { 2024-04-01T13:32:17.521 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:32:17.521 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:32:17.181875+0000" 2024-04-01T13:32:17.521 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.521 INFO:teuthology.orchestra.run.smithi087.stdout: "blocking_event": { 2024-04-01T13:32:17.521 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:32:17.521 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:32:17.521 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.521 INFO:teuthology.orchestra.run.smithi087.stdout: "time_event": { 2024-04-01T13:32:17.521 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:32:17.521 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:32:17.522 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:32:17.522 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:32:17.522 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.522 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:32:17.522 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "client_request", 2024-04-01T13:32:17.522 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 44, 2024-04-01T13:32:17.522 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:32:17.522 INFO:teuthology.orchestra.run.smithi087.stdout: "time_event": { 2024-04-01T13:32:17.522 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:32:17.522 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:32:17.175226+0000" 2024-04-01T13:32:17.522 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.522 INFO:teuthology.orchestra.run.smithi087.stdout: "blocking_event": { 2024-04-01T13:32:17.522 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:32:17.522 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:32:17.175245+0000" 2024-04-01T13:32:17.522 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.523 INFO:teuthology.orchestra.run.smithi087.stdout: "blocking_event": { 2024-04-01T13:32:17.523 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:32:17.523 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:32:17.175424+0000" 2024-04-01T13:32:17.523 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.523 INFO:teuthology.orchestra.run.smithi087.stdout: "blocking_event": { 2024-04-01T13:32:17.523 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:32:17.523 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:32:17.523 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.523 INFO:teuthology.orchestra.run.smithi087.stdout: "blocking_event": { 2024-04-01T13:32:17.523 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:32:17.523 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:32:17.181767+0000" 2024-04-01T13:32:17.523 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.523 INFO:teuthology.orchestra.run.smithi087.stdout: "blocking_event": { 2024-04-01T13:32:17.523 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:32:17.523 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:32:17.182084+0000" 2024-04-01T13:32:17.523 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.523 INFO:teuthology.orchestra.run.smithi087.stdout: "blocking_event": { 2024-04-01T13:32:17.524 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:32:17.524 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:32:17.524 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.524 INFO:teuthology.orchestra.run.smithi087.stdout: "time_event": { 2024-04-01T13:32:17.524 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:32:17.524 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:32:17.524 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:32:17.524 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:32:17.524 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.524 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-01T13:32:17.524 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "client_request", 2024-04-01T13:32:17.524 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 45, 2024-04-01T13:32:17.524 INFO:teuthology.orchestra.run.smithi087.stdout: "detail": { 2024-04-01T13:32:17.524 INFO:teuthology.orchestra.run.smithi087.stdout: "time_event": { 2024-04-01T13:32:17.524 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-01T13:32:17.524 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:32:17.179083+0000" 2024-04-01T13:32:17.525 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.525 INFO:teuthology.orchestra.run.smithi087.stdout: "blocking_event": { 2024-04-01T13:32:17.525 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-01T13:32:17.525 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:32:17.179099+0000" 2024-04-01T13:32:17.525 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.525 INFO:teuthology.orchestra.run.smithi087.stdout: "blocking_event": { 2024-04-01T13:32:17.525 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-01T13:32:17.525 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:32:17.179236+0000" 2024-04-01T13:32:17.525 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.525 INFO:teuthology.orchestra.run.smithi087.stdout: "blocking_event": { 2024-04-01T13:32:17.525 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-01T13:32:17.525 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:32:17.525 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.525 INFO:teuthology.orchestra.run.smithi087.stdout: "blocking_event": { 2024-04-01T13:32:17.525 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-01T13:32:17.525 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:32:17.181974+0000" 2024-04-01T13:32:17.525 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.526 INFO:teuthology.orchestra.run.smithi087.stdout: "blocking_event": { 2024-04-01T13:32:17.526 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-01T13:32:17.526 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-04-01T13:32:17.182168+0000" 2024-04-01T13:32:17.526 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.526 INFO:teuthology.orchestra.run.smithi087.stdout: "blocking_event": { 2024-04-01T13:32:17.526 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-01T13:32:17.526 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:32:17.526 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-01T13:32:17.526 INFO:teuthology.orchestra.run.smithi087.stdout: "time_event": { 2024-04-01T13:32:17.526 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-01T13:32:17.526 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "0.000000" 2024-04-01T13:32:17.526 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:32:17.526 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:32:17.526 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-01T13:32:17.526 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-01T13:32:17.526 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-01T13:32:20.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-01T13:32:20.089 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~54s 2024-04-01T13:32:22.530 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [1, 0, 2] 2024-04-01T13:32:22.530 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-01T13:32:22.530 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-01T13:32:22.531 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-01T13:32:22.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:32:22.666+0000 7f55bb62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:32:22.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:32:22.676+0000 7f55bb62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:32:25.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2024-04-01T13:32:25.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~60s 2024-04-01T13:32:30.695 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~11s 2024-04-01T13:32:30.695 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~65s 2024-04-01T13:32:35.999 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~16s 2024-04-01T13:32:35.999 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~70s 2024-04-01T13:32:41.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~21s 2024-04-01T13:32:41.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~76s 2024-04-01T13:32:46.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~27s 2024-04-01T13:32:46.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~81s 2024-04-01T13:32:51.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~32s 2024-04-01T13:32:51.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~86s 2024-04-01T13:32:57.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~37s 2024-04-01T13:32:57.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~92s 2024-04-01T13:33:02.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~42s 2024-04-01T13:33:02.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~97s 2024-04-01T13:33:07.819 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~48s 2024-04-01T13:33:07.819 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~102s 2024-04-01T13:33:13.122 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~53s 2024-04-01T13:33:13.122 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~107s 2024-04-01T13:33:18.073 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:33:18.087+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:33:18.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~58s 2024-04-01T13:33:18.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~113s 2024-04-01T13:33:23.074 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:33:23.088+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:33:23.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~64s 2024-04-01T13:33:23.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~118s 2024-04-01T13:33:28.074 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:33:28.089+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:33:29.031 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~69s 2024-04-01T13:33:29.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~123s 2024-04-01T13:33:33.075 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:33:33.089+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:33:34.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~74s 2024-04-01T13:33:34.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~129s 2024-04-01T13:33:38.075 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:33:38.090+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:33:39.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~80s 2024-04-01T13:33:39.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~134s 2024-04-01T13:33:43.076 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:33:43.091+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:33:44.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~85s 2024-04-01T13:33:44.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~139s 2024-04-01T13:33:48.077 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:33:48.091+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:33:50.244 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~90s 2024-04-01T13:33:50.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~145s 2024-04-01T13:33:53.077 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:33:53.092+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:33:55.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~95s 2024-04-01T13:33:55.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~150s 2024-04-01T13:33:56.744 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T13:33:56.745 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T13:33:58.078 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:33:58.092+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:33:58.746 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T13:33:58.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:33:58.883+0000 7fd6e0a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:33:58.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:33:58.889+0000 7fd6e0a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:34:00.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~101s 2024-04-01T13:34:00.851 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~155s 2024-04-01T13:34:03.079 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:34:03.093+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:34:06.154 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~106s 2024-04-01T13:34:06.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~161s 2024-04-01T13:34:08.079 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:34:08.093+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:34:11.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~111s 2024-04-01T13:34:11.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~166s 2024-04-01T13:34:13.080 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:34:13.094+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:34:16.762 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~117s 2024-04-01T13:34:16.762 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~171s 2024-04-01T13:34:18.080 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:34:18.094+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:34:22.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~122s 2024-04-01T13:34:22.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~176s 2024-04-01T13:34:22.568 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T13:34:22.568 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-01T13:34:22.569 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-01T13:34:22.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:34:22.703+0000 7f1fcd66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:34:22.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:34:22.713+0000 7f1fcd66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:34:22.704 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-01T13:34:22.707 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-01T13:34:22.707 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-01T13:34:22.708 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-01T13:34:22.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:34:22.830+0000 7fe25cbd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:34:22.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:34:22.840+0000 7fe25cbd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:34:22.845 INFO:teuthology.orchestra.run.smithi059.stderr:no valid command found; 10 closest matches: 2024-04-01T13:34:22.845 INFO:teuthology.orchestra.run.smithi059.stderr:config get 2024-04-01T13:34:22.845 INFO:teuthology.orchestra.run.smithi059.stderr:config help 2024-04-01T13:34:22.845 INFO:teuthology.orchestra.run.smithi059.stderr:config set ... 2024-04-01T13:34:22.845 INFO:teuthology.orchestra.run.smithi059.stderr:config show 2024-04-01T13:34:22.845 INFO:teuthology.orchestra.run.smithi059.stderr:deep_scrub 2024-04-01T13:34:22.845 INFO:teuthology.orchestra.run.smithi059.stderr:dump_historic_ops 2024-04-01T13:34:22.845 INFO:teuthology.orchestra.run.smithi059.stderr:dump_historic_slow_ops 2024-04-01T13:34:22.845 INFO:teuthology.orchestra.run.smithi059.stderr:dump_metrics [] 2024-04-01T13:34:22.845 INFO:teuthology.orchestra.run.smithi059.stderr:dump_ops_in_flight 2024-04-01T13:34:22.845 INFO:teuthology.orchestra.run.smithi059.stderr:dump_pgstate_history 2024-04-01T13:34:22.845 INFO:teuthology.orchestra.run.smithi059.stderr:Error EINVAL: invalid command 2024-04-01T13:34:22.847 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:34:22.848 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-01T13:34:22.848 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-01T13:34:22.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:34:22.970+0000 7fd3f0aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:34:22.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:34:22.980+0000 7fd3f0aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:34:23.081 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:34:23.095+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:34:27.370 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~127s 2024-04-01T13:34:27.370 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~182s 2024-04-01T13:34:28.081 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:34:28.096+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:34:32.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~133s 2024-04-01T13:34:32.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~187s 2024-04-01T13:34:33.082 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:34:33.096+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:34:37.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~138s 2024-04-01T13:34:37.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~192s 2024-04-01T13:34:38.082 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:34:38.097+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:34:43.083 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:34:43.098+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:34:43.282 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~143s 2024-04-01T13:34:43.282 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~198s 2024-04-01T13:34:48.084 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:34:48.098+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:34:48.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~148s 2024-04-01T13:34:48.586 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~203s 2024-04-01T13:34:53.084 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:34:53.099+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:34:53.890 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~154s 2024-04-01T13:34:53.890 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~208s 2024-04-01T13:34:58.085 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:34:58.100+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:34:59.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~159s 2024-04-01T13:34:59.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~214s 2024-04-01T13:35:03.086 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:35:03.101+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:35:04.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~164s 2024-04-01T13:35:04.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~219s 2024-04-01T13:35:08.087 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:35:08.101+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:35:09.800 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~170s 2024-04-01T13:35:09.800 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~224s 2024-04-01T13:35:13.087 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:35:13.102+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:35:15.104 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~175s 2024-04-01T13:35:15.104 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~229s 2024-04-01T13:35:18.087 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:35:18.102+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:35:20.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~180s 2024-04-01T13:35:20.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~235s 2024-04-01T13:35:23.088 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:35:23.102+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:35:25.711 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~186s 2024-04-01T13:35:25.711 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~240s 2024-04-01T13:35:28.089 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:35:28.104+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:35:31.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~191s 2024-04-01T13:35:31.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~245s 2024-04-01T13:35:33.090 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:35:33.104+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:35:36.317 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~196s 2024-04-01T13:35:36.317 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~251s 2024-04-01T13:35:38.090 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:35:38.105+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:35:41.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~202s 2024-04-01T13:35:41.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~256s 2024-04-01T13:35:43.091 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:35:43.106+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:35:46.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~207s 2024-04-01T13:35:46.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~261s 2024-04-01T13:35:48.092 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:35:48.106+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:35:52.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~212s 2024-04-01T13:35:52.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~267s 2024-04-01T13:35:53.092 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:35:53.107+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:35:57.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~217s 2024-04-01T13:35:57.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~272s 2024-04-01T13:35:58.093 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:35:58.108+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:35:58.783 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T13:35:58.784 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T13:36:00.785 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T13:36:00.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:36:00.922+0000 7fa4ee6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:36:00.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:36:00.928+0000 7fa4ee6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:36:02.837 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~223s 2024-04-01T13:36:02.837 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~277s 2024-04-01T13:36:03.094 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:36:03.109+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:36:08.094 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:36:08.109+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:36:08.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~228s 2024-04-01T13:36:08.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~282s 2024-04-01T13:36:13.095 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:36:13.110+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:36:13.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~233s 2024-04-01T13:36:13.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~288s 2024-04-01T13:36:18.096 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:36:18.111+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:36:18.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~239s 2024-04-01T13:36:18.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~293s 2024-04-01T13:36:22.879 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T13:36:22.880 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-01T13:36:23.096 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:36:23.111+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:36:24.050 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~244s 2024-04-01T13:36:24.050 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~298s 2024-04-01T13:36:27.881 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [1, 0, 2] 2024-04-01T13:36:27.881 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-01T13:36:27.881 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-04-01T13:36:27.881 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-01T13:36:27.882 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-01T13:36:27.885 INFO:tasks.ceph.osd.3:Started 2024-04-01T13:36:27.885 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:36:27.997 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:36:27.999 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:36:28.000 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:36:28.097 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:36:28.112+0000 7f7976df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.59:6803/3031929549 for 24sec e45 v45) 2024-04-01T13:36:29.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~249s 2024-04-01T13:36:29.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~304s 2024-04-01T13:36:29.455 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-04-01T13:36:29.858 INFO:tasks.ceph.osd.2:Sent signal 15 2024-04-01T13:36:29.859 INFO:tasks.ceph.osd.3:Sent signal 15 2024-04-01T13:36:29.859 INFO:tasks.ceph.mon.a:Sent signal 15 2024-04-01T13:36:29.859 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-04-01T13:36:29.860 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:36:29.875+0000 7f7979dff640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 29503) UID: 0 2024-04-01T13:36:29.860 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-04-01T13:36:29.875+0000 7f7979dff640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-01T13:36:29.860 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:36:29.875+0000 7fb5003f7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 29525) UID: 0 2024-04-01T13:36:29.861 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-04-01T13:36:29.875+0000 7fb5003f7640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-01T13:36:33.000 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:36:33.100 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:36:33.102 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:36:33.103 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:36:38.104 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:36:38.215 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:36:38.218 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:36:38.218 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:36:43.219 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:36:43.330 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:36:43.332 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:36:43.333 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:36:48.334 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:36:48.446 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:36:48.448 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:36:48.449 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:36:53.450 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:36:53.562 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:36:53.565 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:36:53.565 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:36:58.566 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:36:58.679 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:36:58.681 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:36:58.682 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:37:03.683 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:37:03.795 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:37:03.798 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:37:03.798 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:37:08.799 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:37:08.913 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:37:08.916 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:37:08.916 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:37:13.918 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:37:14.029 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:37:14.032 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:37:14.032 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:37:19.033 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:37:19.145 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:37:19.148 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:37:19.148 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:37:24.150 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:37:24.261 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:37:24.264 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:37:24.264 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:37:29.265 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:37:29.372 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:37:29.375 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:37:29.375 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:37:34.376 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:37:34.488 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:37:34.490 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:37:34.491 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:37:39.492 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:37:39.603 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:37:39.605 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:37:39.606 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:37:44.606 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:37:44.719 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:37:44.721 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:37:44.722 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:37:49.723 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:37:49.834 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:37:49.836 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:37:49.837 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:37:54.838 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:37:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:37:54.951 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:37:54.952 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:37:59.953 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:38:00.065 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:38:00.068 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:38:00.069 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:38:00.824 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T13:38:00.825 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T13:38:02.826 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T13:38:02.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:38:02.963+0000 7fadac79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:38:05.070 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:38:05.175 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:38:05.178 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:38:05.178 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:38:10.180 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:38:10.291 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:38:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:38:10.294 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:38:15.295 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:38:15.407 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:38:15.410 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:38:15.410 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:38:20.412 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:38:20.524 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:38:20.527 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:38:20.527 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:38:25.528 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:38:25.639 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:38:25.641 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:38:25.642 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:38:30.643 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:38:30.753 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:38:30.756 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:38:30.756 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:38:35.757 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:38:35.868 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:38:35.871 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:38:35.871 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:38:40.872 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:38:40.984 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:38:40.986 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:38:40.987 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:38:45.987 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:38:46.099 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:38:46.102 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:38:46.102 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:38:51.106 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:38:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:38:51.219 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:38:51.220 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:38:56.220 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:38:56.331 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:38:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:38:56.334 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:39:01.335 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:39:01.447 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:39:01.450 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:39:01.451 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:39:06.452 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:39:06.565 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:39:06.568 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:39:06.568 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:39:11.570 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:39:11.683 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:39:11.686 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:39:11.687 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:39:16.688 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:39:16.801 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:39:16.804 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:39:16.805 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:39:21.805 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:39:21.913 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:39:21.915 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:39:21.916 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:39:26.917 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:39:27.028 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:39:27.031 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:39:27.032 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:39:32.033 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:39:32.144 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:39:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:39:32.150 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:39:37.150 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:39:37.262 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:39:37.264 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:39:37.265 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:39:42.265 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:39:42.377 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:39:42.379 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:39:42.380 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:39:47.381 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:39:47.492 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:39:47.494 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:39:47.495 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:39:52.496 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:39:52.608 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:39:52.611 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:39:52.611 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:39:57.612 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:39:57.723 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:39:57.726 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:39:57.726 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:40:02.728 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:40:02.841 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:40:02.843 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:40:02.843 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:40:02.867 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T13:40:02.868 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T13:40:04.869 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T13:40:04.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:40:05.000+0000 7f47269f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:40:07.844 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:40:07.955 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:40:07.958 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:40:07.958 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:40:12.959 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:40:13.070 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:40:13.072 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:40:13.073 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:40:18.074 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:40:18.185 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:40:18.187 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:40:18.188 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:40:23.189 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:40:23.301 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:40:23.304 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:40:23.304 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:40:28.306 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:40:28.416 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:40:28.419 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:40:28.420 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:40:33.421 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:40:33.531 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:40:33.533 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:40:33.534 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:40:38.534 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:40:38.646 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:40:38.648 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:40:38.649 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:40:43.650 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:40:43.761 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:40:43.764 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:40:43.764 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:40:48.766 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:40:48.877 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:40:48.880 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:40:48.880 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:40:53.880 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:40:53.993 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:40:53.996 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:40:53.996 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:40:58.997 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:40:59.110 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:40:59.112 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:40:59.113 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:41:04.114 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:41:04.225 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:41:04.227 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:41:04.228 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:41:09.229 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:41:09.340 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:41:09.343 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:41:09.343 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:41:14.345 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:41:14.457 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:41:14.459 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:41:14.460 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:41:19.461 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:41:19.573 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:41:19.575 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:41:19.576 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:41:24.577 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:41:24.688 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:41:24.691 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:41:24.691 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:41:29.693 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:41:29.805 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:41:29.807 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:41:29.808 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:41:34.809 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:41:34.920 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:41:34.922 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:41:34.923 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:41:39.924 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:41:40.035 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:41:40.038 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:41:40.038 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:41:45.039 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:41:45.150 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:41:45.152 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:41:45.153 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:41:50.154 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:41:50.265 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:41:50.268 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:41:50.268 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:41:55.269 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:41:55.379 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:41:55.382 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:41:55.382 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:42:00.383 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:42:00.494 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:42:00.496 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:42:00.497 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:42:04.909 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T13:42:04.909 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T13:42:05.498 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:42:05.609 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:42:05.611 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:42:05.612 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:42:06.910 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T13:42:07.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:42:07.039+0000 7f79f8cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:42:10.613 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:42:10.724 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:42:10.726 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:42:10.727 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:42:15.727 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:42:15.839 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:42:15.842 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:42:15.843 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:42:20.844 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:42:20.957 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:42:20.960 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:42:20.960 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:42:25.961 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:42:26.074 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:42:26.078 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:42:26.078 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:42:31.079 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:42:31.190 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:42:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:42:31.193 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-01T13:42:36.195 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-01T13:42:36.306 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-01T13:42:36.309 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-01T13:42:36.311 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_afa73a9e21b8fb0610f8af7266f1975be37c0a69/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph-c_afa73a9e21b8fb0610f8af7266f1975be37c0a69/qa/tasks/ceph_manager.py", line 1461, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph-c_afa73a9e21b8fb0610f8af7266f1975be37c0a69/qa/tasks/ceph_manager.py", line 560, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_ceph_ceph-c_afa73a9e21b8fb0610f8af7266f1975be37c0a69/qa/tasks/ceph_manager.py", line 3122, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_ceph_ceph-c_afa73a9e21b8fb0610f8af7266f1975be37c0a69/qa/tasks/ceph_manager.py", line 2028, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.3 restart 2024-04-01T13:42:36.311 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_afa73a9e21b8fb0610f8af7266f1975be37c0a69/qa/tasks/ceph_manager.py", line 1332, in do_thrash self._do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_afa73a9e21b8fb0610f8af7266f1975be37c0a69/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph-c_afa73a9e21b8fb0610f8af7266f1975be37c0a69/qa/tasks/ceph_manager.py", line 1461, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph-c_afa73a9e21b8fb0610f8af7266f1975be37c0a69/qa/tasks/ceph_manager.py", line 560, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_ceph_ceph-c_afa73a9e21b8fb0610f8af7266f1975be37c0a69/qa/tasks/ceph_manager.py", line 3122, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_ceph_ceph-c_afa73a9e21b8fb0610f8af7266f1975be37c0a69/qa/tasks/ceph_manager.py", line 2028, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.3 restart 2024-04-01T13:44:06.950 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T13:44:06.951 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T13:44:08.953 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T13:44:09.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:44:09.080+0000 7f5de195d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:46:08.992 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T13:46:08.993 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T13:46:10.994 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T13:46:11.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:46:11.120+0000 7f4ffcf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:48:11.034 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T13:48:11.035 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T13:48:13.035 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T13:48:13.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:48:13.159+0000 7efd0d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:50:13.074 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T13:50:13.075 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T13:50:15.077 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T13:50:15.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:50:15.199+0000 7f94d3e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:52:15.115 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T13:52:15.116 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T13:52:17.117 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T13:52:17.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:52:17.240+0000 7fede2910640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:54:17.157 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T13:54:17.158 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T13:54:19.159 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T13:54:19.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:54:19.279+0000 7f43ab9d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:56:19.198 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T13:56:19.199 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T13:56:21.200 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T13:56:21.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:56:21.317+0000 7fa9d54ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T13:58:21.239 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T13:58:21.240 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T13:58:23.241 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T13:58:23.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T13:58:23.361+0000 7f202eb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:00:23.279 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:00:23.280 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:00:25.281 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T14:00:25.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:00:25.400+0000 7f9868e24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:02:25.320 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:02:25.321 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:02:27.323 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T14:02:27.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:02:27.442+0000 7f6cde66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:04:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:04:27.363 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:04:29.364 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T14:04:29.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:04:29.483+0000 7f396bd1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:06:29.403 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:06:29.404 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:06:31.405 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T14:06:31.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:06:31.522+0000 7ff1ab19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:08:31.444 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:08:31.445 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:08:33.446 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T14:08:33.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:08:33.564+0000 7f1b7d2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:10:33.484 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:10:33.486 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:10:35.487 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T14:10:35.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:10:35.603+0000 7f91e586d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:12:35.526 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:12:35.527 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:12:37.528 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T14:12:37.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:12:37.644+0000 7f60dd4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:14:37.565 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:14:37.566 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:14:39.567 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T14:14:39.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:14:39.683+0000 7ff0298ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:16:39.604 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:16:39.605 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:16:41.606 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T14:16:41.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:16:41.724+0000 7f7fee8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:18:41.644 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:18:41.645 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:18:43.646 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T14:18:43.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:18:43.764+0000 7f7c280ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:20:43.683 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:20:43.684 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:20:45.686 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T14:20:45.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:20:45.804+0000 7fc3c26ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:22:45.723 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:22:45.725 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:22:47.726 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T14:22:47.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:22:47.844+0000 7f7f0aaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:24:47.764 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:24:47.765 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:24:49.766 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T14:24:49.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:24:49.885+0000 7f2a254ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:26:49.805 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:26:49.806 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:26:51.807 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T14:26:51.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:26:51.926+0000 7feaf3aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:28:51.845 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:28:51.846 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:28:53.847 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T14:28:53.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:28:53.966+0000 7f9ebf39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:30:53.885 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:30:53.890 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:30:55.891 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T14:30:56.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:30:56.009+0000 7f10c339d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:32:55.929 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:32:55.930 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:32:57.932 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T14:32:58.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:32:58.051+0000 7f0ab182d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:34:57.970 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:34:57.971 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:34:59.972 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T14:35:00.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:35:00.091+0000 7fd41611d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:37:00.013 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:37:00.015 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:37:02.016 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T14:37:02.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:37:02.134+0000 7fb1d78ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:39:02.054 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:39:02.055 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:39:04.056 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T14:39:04.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:39:04.174+0000 7f872d62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:41:04.095 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:41:04.096 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:41:06.097 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T14:41:06.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:41:06.217+0000 7fd77cbe7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:43:06.135 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:43:06.136 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:43:08.137 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T14:43:08.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:43:08.256+0000 7f69b599d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:45:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:45:08.176 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:45:10.177 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T14:45:10.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:45:10.295+0000 7f83b8b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:47:10.214 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:47:10.215 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:47:12.216 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T14:47:12.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:47:12.334+0000 7f8e9366d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:49:12.253 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:49:12.254 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:49:14.255 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T14:49:14.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:49:14.376+0000 7fa3c939d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:51:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:51:14.294 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:51:16.295 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T14:51:16.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:51:16.415+0000 7f826acad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:53:16.333 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:53:16.334 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:53:18.336 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T14:53:18.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:53:18.456+0000 7f7a0e22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:55:18.375 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:55:18.375 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:55:20.377 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T14:55:20.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:55:20.497+0000 7f6f1e06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:57:20.415 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:57:20.416 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:57:22.416 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T14:57:22.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:57:22.539+0000 7f31b7754640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T14:59:22.454 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T14:59:22.455 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T14:59:24.456 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T14:59:24.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T14:59:24.576+0000 7f7dc8b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:01:24.494 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:01:24.495 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:01:26.496 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T15:01:26.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:01:26.616+0000 7fcad2064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:03:26.533 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:03:26.534 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:03:28.535 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T15:03:28.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:03:28.658+0000 7f1a56f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:05:28.574 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:05:28.575 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:05:30.576 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T15:05:30.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:05:30.697+0000 7f5d8595d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:07:30.614 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:07:30.615 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:07:32.617 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T15:07:32.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:07:32.736+0000 7f6075cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:09:32.654 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:09:32.655 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:09:34.656 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T15:09:34.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:09:34.777+0000 7f3df9f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:11:34.694 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:11:34.695 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:11:36.696 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T15:11:36.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:11:36.818+0000 7f97860ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:13:36.734 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:13:36.735 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:13:38.736 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T15:13:38.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:13:38.855+0000 7fca3c31d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:15:38.773 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:15:38.774 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:15:40.776 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T15:15:40.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:15:40.897+0000 7fe41379d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:17:40.814 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:17:40.815 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:17:42.816 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T15:17:42.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:17:42.936+0000 7ff2b7b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:19:42.855 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:19:42.855 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:19:44.857 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T15:19:44.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:19:44.977+0000 7fe8fb55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:21:44.894 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:21:44.895 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:21:46.896 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T15:21:47.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:21:47.016+0000 7f452d35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:23:46.933 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:23:46.934 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:23:48.935 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T15:23:49.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:23:49.058+0000 7f2e24114640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:25:48.972 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:25:48.973 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:25:50.975 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T15:25:51.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:25:51.095+0000 7f7dabaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:27:51.014 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:27:51.015 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:27:53.016 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T15:27:53.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:27:53.137+0000 7fe5ef99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:29:53.054 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:29:53.055 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:29:55.056 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T15:29:55.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:29:55.177+0000 7fe90986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:31:55.093 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:31:55.094 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:31:57.096 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T15:31:57.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:31:57.218+0000 7fa234824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:33:57.134 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:33:57.135 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:33:59.136 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T15:33:59.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:33:59.258+0000 7f3c25354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:35:59.174 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:35:59.175 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:36:01.176 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T15:36:01.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:36:01.297+0000 7fb368514640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:38:01.213 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:38:01.214 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:38:03.215 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T15:38:03.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:38:03.339+0000 7f4184e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:40:03.271 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:40:03.272 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:40:05.273 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T15:40:05.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:40:05.395+0000 7f5a2e4ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:42:05.311 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:42:05.328 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:42:07.330 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T15:42:07.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:42:07.452+0000 7ffb306ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:44:07.368 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:44:07.369 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:44:09.370 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T15:44:09.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:44:09.491+0000 7fc13d0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:46:09.407 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:46:09.408 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:46:11.409 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T15:46:11.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:46:11.532+0000 7f7e6199d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:48:11.447 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:48:11.448 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:48:13.449 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T15:48:13.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:48:13.572+0000 7fc21795d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:50:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:50:13.488 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:50:15.488 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T15:50:15.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:50:15.611+0000 7f150d79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:52:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:52:15.550 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:52:17.552 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T15:52:17.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:52:17.672+0000 7f138d6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:54:17.589 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:54:17.590 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:54:19.592 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T15:54:19.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:54:19.713+0000 7f1ebec6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:56:19.629 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:56:19.630 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:56:21.631 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T15:56:21.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:56:21.755+0000 7f7ae0f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T15:58:21.668 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T15:58:21.669 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T15:58:23.670 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T15:58:23.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T15:58:23.792+0000 7f245ba06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:00:23.709 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:00:23.709 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:00:25.710 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T16:00:25.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:00:25.832+0000 7fc2ec66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:02:25.748 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:02:25.749 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:02:27.750 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T16:02:27.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:02:27.872+0000 7f2eac31d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:04:27.788 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:04:27.789 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:04:29.790 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T16:04:29.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:04:29.912+0000 7f3cb0e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:06:29.828 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:06:29.829 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:06:31.830 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T16:06:31.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:06:31.952+0000 7fa0d799d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:08:31.868 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:08:31.869 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:08:33.870 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T16:08:33.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:08:33.992+0000 7fb5ff39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:10:33.908 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:10:33.909 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:10:35.910 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T16:10:36.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:10:36.032+0000 7f808e46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:12:35.948 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:12:35.949 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:12:37.950 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T16:12:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:12:38.072+0000 7f3bb982d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:14:37.987 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:14:37.988 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:14:39.989 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T16:14:40.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:14:40.106+0000 7f76004ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:16:40.025 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:16:40.026 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:16:42.027 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T16:16:42.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:16:42.149+0000 7f160db5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:18:42.065 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:18:42.066 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:18:44.069 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T16:18:44.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:18:44.191+0000 7f7ec9cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:20:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:20:44.108 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:20:46.109 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T16:20:46.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:20:46.231+0000 7f08c94ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:22:46.147 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:22:46.148 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:22:48.150 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T16:22:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:22:48.271+0000 7f3c5422d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:24:48.187 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:24:48.188 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:24:50.189 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T16:24:50.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:24:50.311+0000 7fa03011d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:26:50.226 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:26:50.227 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:26:52.229 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T16:26:52.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:26:52.352+0000 7f214815d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:28:52.267 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:28:52.268 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:28:54.270 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T16:28:54.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:28:54.392+0000 7fc0ae51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:30:54.308 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:30:54.309 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:30:56.310 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T16:30:56.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:30:56.433+0000 7f276ff5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:32:56.348 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:32:56.349 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:32:58.350 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T16:32:58.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:32:58.472+0000 7fac8d4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:34:58.388 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:34:58.389 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:35:00.391 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T16:35:00.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:35:00.512+0000 7f7da23f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:37:00.429 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:37:00.431 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:37:02.432 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T16:37:02.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:37:02.554+0000 7f8af7c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:39:02.470 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:39:02.471 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:39:04.472 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T16:39:04.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:39:04.594+0000 7fd148813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:41:04.510 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:41:04.511 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:41:06.513 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T16:41:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:41:06.636+0000 7f152ed1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:43:06.551 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:43:06.552 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:43:08.553 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T16:43:08.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:43:08.675+0000 7f950831d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:45:08.591 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:45:08.592 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:45:10.593 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T16:45:10.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:45:10.715+0000 7f11fe0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:47:10.631 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:47:10.632 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:47:12.633 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T16:47:12.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:47:12.756+0000 7f753bc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:49:12.671 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:49:12.673 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:49:14.674 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T16:49:14.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:49:14.796+0000 7f29704ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:51:14.712 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:51:14.713 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:51:16.714 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T16:51:16.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:51:16.835+0000 7fd12b1f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:53:16.752 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:53:16.753 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:53:18.754 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T16:53:18.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:53:18.876+0000 7fb3a959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:55:18.792 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:55:18.793 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:55:20.794 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T16:55:20.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:55:20.916+0000 7fea89f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:57:20.832 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:57:20.833 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:57:22.834 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T16:57:22.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:57:22.955+0000 7f90e3e17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T16:59:22.872 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T16:59:22.873 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T16:59:24.875 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T16:59:24.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T16:59:24.996+0000 7f6ea726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:01:24.913 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:01:24.913 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:01:26.915 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T17:01:27.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:01:27.037+0000 7f4c8f35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:03:26.953 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:03:26.954 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:03:28.955 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T17:03:29.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:03:29.078+0000 7f554d99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:05:28.993 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:05:28.994 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:05:30.995 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T17:05:31.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:05:31.116+0000 7f306c424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:07:31.033 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:07:31.034 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:07:33.035 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T17:07:33.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:07:33.157+0000 7fbcba617640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:09:33.073 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:09:33.074 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:09:35.076 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T17:09:35.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:09:35.197+0000 7fc411c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:11:35.113 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:11:35.114 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:11:37.116 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T17:11:37.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:11:37.238+0000 7f937a6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:13:37.154 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:13:37.155 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:13:39.156 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T17:13:39.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:13:39.278+0000 7f914406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:15:39.195 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:15:39.196 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:15:41.197 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T17:15:41.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:15:41.318+0000 7fe349417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:17:41.236 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:17:41.237 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:17:43.238 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T17:17:43.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:17:43.359+0000 7f4bd846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:19:43.275 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:19:43.276 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:19:45.277 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T17:19:45.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:19:45.399+0000 7f524d95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:21:45.316 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:21:45.317 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:21:47.317 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T17:21:47.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:21:47.439+0000 7fa80b82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:23:47.355 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:23:47.356 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:23:49.357 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T17:23:49.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:23:49.479+0000 7ff3f626d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:25:49.395 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:25:49.396 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:25:51.398 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T17:25:51.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:25:51.519+0000 7f9f8fb14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:27:51.436 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:27:51.437 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:27:53.438 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T17:27:53.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:27:53.560+0000 7ff09f013640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:29:53.476 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:29:53.477 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:29:55.479 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T17:29:55.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:29:55.601+0000 7f96bb79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:31:55.517 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:31:55.518 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:31:57.519 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T17:31:57.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:31:57.640+0000 7f03d959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:33:57.558 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:33:57.559 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:33:59.560 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T17:33:59.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:33:59.681+0000 7fadd946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:35:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:35:59.599 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:36:01.600 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T17:36:01.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:36:01.722+0000 7ffa31a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:38:01.639 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:38:01.640 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:38:03.641 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T17:38:03.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:38:03.765+0000 7f88efe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:40:03.679 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:40:03.680 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:40:05.682 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T17:40:05.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:40:05.803+0000 7fd7c686d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:42:05.720 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:42:05.721 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:42:07.722 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T17:42:07.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:42:07.845+0000 7fb57a006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:44:07.760 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:44:07.760 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:44:09.761 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T17:44:09.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:44:09.883+0000 7fa963f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:46:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:46:09.801 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:46:11.802 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T17:46:11.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:46:11.924+0000 7f0662e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:48:11.843 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:48:11.844 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:48:13.844 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T17:48:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:48:13.967+0000 7f093f51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:50:13.882 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:50:13.883 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:50:15.884 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T17:50:16.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:50:16.006+0000 7f8cd1660640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:52:15.922 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:52:15.923 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:52:17.924 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T17:52:18.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:52:18.046+0000 7fd0f46ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:54:17.962 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:54:17.962 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:54:19.963 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T17:54:20.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:54:20.086+0000 7fdb9ac6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:56:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:56:20.003 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:56:22.004 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T17:56:22.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:56:22.125+0000 7f9b8d39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T17:58:22.042 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T17:58:22.043 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T17:58:24.045 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T17:58:24.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T17:58:24.167+0000 7fd60bc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:00:24.083 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:00:24.084 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:00:26.085 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T18:00:26.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:00:26.207+0000 7fade10ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:02:26.123 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:02:26.124 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:02:28.126 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T18:02:28.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:02:28.248+0000 7f5726606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:04:28.163 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:04:28.164 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:04:30.165 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T18:04:30.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:04:30.287+0000 7f9a8aa20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:06:30.203 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:06:30.204 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:06:32.206 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T18:06:32.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:06:32.328+0000 7f281966d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:08:32.243 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:08:32.244 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:08:34.245 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T18:08:34.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:08:34.368+0000 7f078ef1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:10:34.283 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:10:34.283 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:10:36.284 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T18:10:36.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:10:36.407+0000 7fe10e754640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:12:36.322 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:12:36.322 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:12:38.324 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T18:12:38.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:12:38.447+0000 7f3ec606d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:14:38.362 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:14:38.363 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:14:40.364 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T18:14:40.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:14:40.488+0000 7f354d9f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:16:40.402 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:16:40.403 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:16:42.404 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T18:16:42.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:16:42.527+0000 7f14fc660640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:18:42.443 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:18:42.444 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:18:44.445 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T18:18:44.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:18:44.568+0000 7f3e2c06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:20:44.484 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:20:44.485 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:20:46.486 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T18:20:46.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:20:46.608+0000 7fae40bff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:22:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:22:46.525 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:22:48.526 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T18:22:48.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:22:48.648+0000 7f013962d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:24:48.564 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:24:48.565 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:24:50.567 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T18:24:50.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:24:50.690+0000 7fc42e55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:26:50.605 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:26:50.607 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:26:52.608 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T18:26:52.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:26:52.730+0000 7f5e70a24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:28:52.646 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:28:52.647 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:28:54.649 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T18:28:54.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:28:54.772+0000 7f728915d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:30:54.687 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:30:54.688 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:30:56.689 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T18:30:56.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:30:56.813+0000 7f0217b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:32:56.727 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:32:56.728 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:32:58.730 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T18:32:58.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:32:58.854+0000 7f5f5399d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:34:58.768 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:34:58.769 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:35:00.770 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T18:35:00.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:35:00.892+0000 7f54d61d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:37:00.808 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:37:00.809 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:37:02.811 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T18:37:02.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:37:02.933+0000 7f103dead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:39:02.849 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:39:02.850 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:39:04.851 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T18:39:04.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:39:04.973+0000 7f38a7f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:41:04.890 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:41:04.891 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:41:06.892 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T18:41:07.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:41:07.017+0000 7f8c92f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:43:06.931 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:43:06.932 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:43:08.933 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T18:43:09.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:43:09.056+0000 7fa13e19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:45:08.971 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:45:08.972 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:45:10.973 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T18:45:11.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:45:11.096+0000 7fd480d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:47:11.011 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:47:11.012 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:47:13.013 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T18:47:13.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:47:13.134+0000 7f0d71a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:49:13.050 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:49:13.051 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:49:15.053 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T18:49:15.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:49:15.175+0000 7f665c0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:51:15.091 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:51:15.092 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:51:17.094 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T18:51:17.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:51:17.216+0000 7f33ffe2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:53:17.132 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:53:17.133 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:53:19.134 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T18:53:19.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:53:19.257+0000 7f449655d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:55:19.173 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:55:19.174 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:55:21.175 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T18:55:21.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:55:21.298+0000 7f2d7771d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:57:21.213 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:57:21.215 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:57:23.215 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T18:57:23.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:57:23.339+0000 7fa65491d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T18:59:23.254 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T18:59:23.255 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T18:59:25.257 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T18:59:25.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T18:59:25.381+0000 7fea64e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:01:25.295 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:01:25.296 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:01:27.297 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T19:01:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:01:27.419+0000 7f4d0c0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:03:27.335 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:03:27.336 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:03:29.337 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T19:03:29.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:03:29.460+0000 7f2d7326d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:05:29.375 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:05:29.376 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:05:31.377 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T19:05:31.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:05:31.498+0000 7f3c91a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:07:31.416 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:07:31.417 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:07:33.419 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T19:07:33.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:07:33.540+0000 7fa38b1d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:09:33.457 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:09:33.458 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:09:35.459 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T19:09:35.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:09:35.581+0000 7f666786d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:11:35.496 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:11:35.497 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:11:37.498 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T19:11:37.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:11:37.620+0000 7f467be24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:13:37.535 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:13:37.536 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:13:39.537 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T19:13:39.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:13:39.658+0000 7f511ae64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:15:39.575 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:15:39.576 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:15:41.577 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T19:15:41.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:15:41.699+0000 7f5d586ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:17:41.615 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:17:41.616 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:17:43.617 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T19:17:43.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:17:43.740+0000 7f8b9859d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:19:43.655 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:19:43.656 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:19:45.657 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T19:19:45.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:19:45.779+0000 7fba78eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:21:45.696 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:21:45.697 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:21:47.699 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T19:21:47.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:21:47.823+0000 7f704c9fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:23:47.739 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:23:47.740 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:23:49.741 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T19:23:49.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:23:49.865+0000 7fcfa1a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:25:49.780 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:25:49.781 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:25:51.783 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T19:25:51.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:25:51.906+0000 7f4cecd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:27:51.820 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:27:51.821 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:27:53.823 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T19:27:53.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:27:53.945+0000 7f3d6966d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:29:53.860 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:29:53.861 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:29:55.862 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T19:29:55.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:29:55.985+0000 7f10f575d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:31:55.901 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:31:55.902 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:31:57.902 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T19:31:58.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:31:58.026+0000 7f2f73e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:33:57.941 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:33:57.942 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:33:59.943 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T19:34:00.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:34:00.066+0000 7fac3e8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:35:59.980 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:35:59.981 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:36:01.983 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T19:36:02.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:36:02.106+0000 7fb4bd02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:38:02.021 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:38:02.021 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:38:04.023 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T19:38:04.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:38:04.145+0000 7febcf06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:40:04.060 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:40:04.061 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:40:06.062 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T19:40:06.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:40:06.186+0000 7fb9edfd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:42:06.100 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:42:06.101 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:42:08.102 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T19:42:08.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:42:08.228+0000 7f9d0762d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:44:08.140 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:44:08.141 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:44:10.142 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T19:44:10.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:44:10.266+0000 7f7c2155d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:46:10.180 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:46:10.181 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:46:12.182 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T19:46:12.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:46:12.305+0000 7fc510aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:48:12.220 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:48:12.221 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:48:14.222 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T19:48:14.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:48:14.345+0000 7fb3fb0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:50:14.260 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:50:14.261 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:50:16.262 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T19:50:16.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:50:16.386+0000 7f069aa64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:52:16.300 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:52:16.301 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:52:18.302 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T19:52:18.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:52:18.425+0000 7f92b7e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:54:18.340 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:54:18.341 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:54:20.342 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T19:54:20.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:54:20.465+0000 7fdb9d464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:56:20.381 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:56:20.382 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:56:22.382 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T19:56:22.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:56:22.506+0000 7f91afc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T19:58:22.421 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T19:58:22.422 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T19:58:24.423 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T19:58:24.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T19:58:24.546+0000 7f58f515d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:00:24.461 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:00:24.461 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:00:26.463 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T20:00:26.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:00:26.585+0000 7f45a942d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:02:26.501 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:02:26.501 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:02:28.502 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T20:02:28.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:02:28.625+0000 7f330519d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:04:28.541 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:04:28.542 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:04:30.544 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T20:04:30.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:04:30.666+0000 7fab640ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:06:30.582 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:06:30.583 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:06:32.586 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T20:06:32.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:06:32.709+0000 7fa74079d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:08:32.624 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:08:32.625 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:08:34.627 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T20:08:34.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:08:34.749+0000 7fee8c75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:10:34.665 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:10:34.666 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:10:36.667 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T20:10:36.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:10:36.789+0000 7f2603aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:12:36.706 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:12:36.707 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:12:38.707 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T20:12:38.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:12:38.830+0000 7f4a4099d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:14:38.746 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:14:38.747 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:14:40.748 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T20:14:40.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:14:40.870+0000 7f3a75ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:16:40.786 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:16:40.787 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:16:42.788 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T20:16:42.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:16:42.910+0000 7f890d42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:18:42.826 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:18:42.827 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:18:44.829 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T20:18:44.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:18:44.950+0000 7f36ab22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:20:44.867 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:20:44.868 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:20:46.869 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T20:20:46.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:20:46.991+0000 7fd173d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:22:46.907 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:22:46.908 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:22:48.909 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T20:22:49.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:22:49.032+0000 7f17b4aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:24:48.947 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:24:48.948 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:24:50.949 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T20:24:51.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:24:51.071+0000 7fb44299d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:26:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:26:50.989 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:26:52.990 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T20:26:53.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:26:53.114+0000 7fd24691d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:28:53.029 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:28:53.031 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:28:55.032 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T20:28:55.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:28:55.153+0000 7f6acf55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:30:55.069 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:30:55.070 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:30:57.071 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T20:30:57.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:30:57.194+0000 7f1c7f217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:32:57.110 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:32:57.111 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:32:59.112 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T20:32:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:32:59.234+0000 7f726386d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:34:59.150 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:34:59.151 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:35:01.152 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T20:35:01.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:35:01.273+0000 7f44163e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:37:01.191 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:37:01.192 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:37:03.193 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T20:37:03.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:37:03.313+0000 7f6851aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:39:03.231 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:39:03.232 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:39:05.233 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T20:39:05.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:39:05.355+0000 7f0f3d910640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:41:05.271 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:41:05.272 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:41:07.273 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T20:41:07.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:41:07.394+0000 7f7956d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:43:07.311 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:43:07.312 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:43:09.313 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T20:43:09.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:43:09.434+0000 7f31101d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:45:09.351 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:45:09.352 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:45:11.353 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T20:45:11.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:45:11.474+0000 7f16f24ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:47:11.392 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:47:11.413 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:47:13.415 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T20:47:13.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:47:13.536+0000 7f49e302d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:49:13.453 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:49:13.454 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:49:15.455 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T20:49:15.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:49:15.576+0000 7faa64eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:51:15.494 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:51:15.494 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:51:17.496 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T20:51:17.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:51:17.618+0000 7f153155d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:53:17.535 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:53:17.536 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:53:19.537 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T20:53:19.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:53:19.657+0000 7f426191d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:55:19.575 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:55:19.591 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:55:21.592 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T20:55:21.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:55:21.713+0000 7fe4cd86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:57:21.630 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:57:21.631 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:57:23.632 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T20:57:23.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:57:23.754+0000 7f89ed31d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T20:59:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T20:59:23.672 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T20:59:25.673 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T20:59:25.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T20:59:25.793+0000 7f4cce82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:01:25.711 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:01:25.712 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:01:27.713 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T21:01:27.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:01:27.832+0000 7f520959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:03:27.750 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:03:27.751 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:03:29.752 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T21:03:29.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:03:29.873+0000 7f8d2846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:05:29.790 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:05:29.791 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:05:31.792 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T21:05:31.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:05:31.913+0000 7f6b28a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:07:31.830 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:07:31.831 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:07:33.832 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T21:07:33.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:07:33.954+0000 7f5f12554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:09:33.871 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:09:33.872 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:09:35.874 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T21:09:35.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:09:35.994+0000 7f9911a17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:11:35.911 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:11:35.912 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:11:37.914 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T21:11:38.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:11:38.035+0000 7f5bc4e20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:13:37.952 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:13:37.954 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:13:39.957 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T21:13:40.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:13:40.078+0000 7f5ada4ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:15:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:15:39.996 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:15:41.997 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T21:15:42.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:15:42.118+0000 7fe8f74ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:17:42.035 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:17:42.036 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:17:44.038 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T21:17:44.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:17:44.158+0000 7f7121c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:19:44.076 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:19:44.077 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:19:46.079 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T21:19:46.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:19:46.200+0000 7f150335d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:21:46.117 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:21:46.118 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:21:48.119 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T21:21:48.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:21:48.240+0000 7efc60e13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:23:48.157 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:23:48.158 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:23:50.159 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T21:23:50.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:23:50.282+0000 7f828566d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:25:50.197 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:25:50.198 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:25:52.199 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T21:25:52.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:25:52.321+0000 7fde869d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:27:52.237 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:27:52.238 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:27:54.239 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T21:27:54.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:27:54.360+0000 7fc9b66ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:29:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:29:54.278 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:29:56.280 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T21:29:56.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:29:56.401+0000 7f81a9aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:31:56.319 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:31:56.320 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:31:58.321 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T21:31:58.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:31:58.443+0000 7f4cadb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:33:58.359 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:33:58.360 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:34:00.362 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T21:34:00.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:34:00.482+0000 7f0061f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:36:00.401 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:36:00.402 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:36:02.403 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T21:36:02.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:36:02.524+0000 7f1a8715d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:38:02.441 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:38:02.442 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:38:04.444 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T21:38:04.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:38:04.565+0000 7effc742d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:40:04.482 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:40:04.483 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:40:06.484 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T21:40:06.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:40:06.605+0000 7f2e5a4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:42:06.523 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:42:06.524 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:42:08.525 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T21:42:08.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:42:08.646+0000 7f45e6606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:44:08.563 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:44:08.564 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:44:10.565 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T21:44:10.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:44:10.686+0000 7fe83ba24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:46:10.603 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:46:10.604 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:46:12.605 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T21:46:12.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:46:12.726+0000 7f9dbe86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:48:12.643 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:48:12.644 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:48:14.645 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T21:48:14.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:48:14.765+0000 7f15e4e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:50:14.683 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:50:14.684 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:50:16.684 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T21:50:16.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:50:16.805+0000 7fda6ba6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:52:16.722 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:52:16.723 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:52:18.724 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T21:52:18.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:52:18.844+0000 7f3e04424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:54:18.762 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:54:18.763 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:54:20.765 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T21:54:20.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:54:20.886+0000 7ff3de820640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:56:20.803 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:56:20.804 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:56:22.805 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T21:56:22.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:56:22.927+0000 7f910e06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T21:58:22.844 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T21:58:22.844 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T21:58:24.845 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T21:58:24.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T21:58:24.968+0000 7f367b75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:00:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:00:24.885 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:00:26.885 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T22:00:27.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:00:27.006+0000 7f4ee5c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:02:26.923 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:02:26.924 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:02:28.925 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T22:02:29.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:02:29.052+0000 7f5a33f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:04:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:04:28.967 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:04:30.968 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T22:04:31.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:04:31.089+0000 7efe9482d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:06:31.007 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:06:31.008 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:06:33.009 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T22:06:33.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:06:33.130+0000 7f0ac9c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:08:33.047 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:08:33.048 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:08:35.049 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T22:08:35.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:08:35.172+0000 7ff9ccc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:10:35.088 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:10:35.089 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:10:37.090 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T22:10:37.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:10:37.212+0000 7f26d919d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:12:37.128 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:12:37.129 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:12:39.130 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T22:12:39.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:12:39.250+0000 7fe6e886d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:14:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:14:39.169 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:14:41.170 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T22:14:41.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:14:41.292+0000 7f05b019d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:16:41.208 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:16:41.209 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:16:43.210 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T22:16:43.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:16:43.332+0000 7fe8832ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:18:43.249 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:18:43.250 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:18:45.251 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T22:18:45.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:18:45.373+0000 7fd72e75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:20:45.289 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:20:45.290 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:20:47.291 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T22:20:47.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:20:47.412+0000 7f1539b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:22:47.328 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:22:47.329 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:22:49.330 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T22:22:49.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:22:49.454+0000 7ff3f219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:24:49.371 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:24:49.372 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:24:51.372 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T22:24:51.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:24:51.494+0000 7f9ecec13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:26:51.411 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:26:51.411 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:26:53.413 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T22:26:53.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:26:53.535+0000 7f573aa24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:28:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:28:53.452 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:28:55.454 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T22:28:55.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:28:55.575+0000 7f61e346d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:30:55.492 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:30:55.493 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:30:57.495 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T22:30:57.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:30:57.616+0000 7efcceced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:32:57.533 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:32:57.534 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:32:59.535 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T22:32:59.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:32:59.656+0000 7f81ece2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:34:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:34:59.574 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:35:01.576 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T22:35:01.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:35:01.698+0000 7f49ebe24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:37:01.614 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:37:01.615 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:37:03.616 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T22:37:03.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:37:03.738+0000 7fcaef6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:39:03.654 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:39:03.655 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:39:05.656 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T22:39:05.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:39:05.778+0000 7f13e355d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:41:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:41:05.695 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:41:07.696 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T22:41:07.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:41:07.818+0000 7fa69ff5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:43:07.734 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:43:07.735 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:43:09.736 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T22:43:09.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:43:09.859+0000 7fd26015d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:45:09.773 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:45:09.774 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:45:11.775 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T22:45:11.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:45:11.897+0000 7f94297f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:47:11.812 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:47:11.813 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:47:13.814 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T22:47:13.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:47:13.937+0000 7f70e1c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:49:13.851 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:49:13.852 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:49:15.853 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T22:49:15.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:49:15.976+0000 7f300d6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:51:15.891 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:51:15.891 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:51:17.894 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T22:51:18.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:51:18.019+0000 7fb811d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:53:17.931 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:53:17.933 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:53:19.934 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T22:53:20.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:53:20.058+0000 7ff12baad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:55:19.971 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:55:19.972 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:55:21.974 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T22:55:22.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:55:22.100+0000 7fe2ad95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:57:22.012 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:57:22.013 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:57:24.015 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T22:57:24.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:57:24.140+0000 7f61c2420640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T22:59:24.052 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T22:59:24.053 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T22:59:26.054 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T22:59:26.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T22:59:26.180+0000 7f62b1be7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:01:26.092 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:01:26.114 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:01:28.116 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T23:01:28.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:01:28.242+0000 7ffab3cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:03:28.154 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:03:28.155 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:03:30.156 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T23:03:30.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:03:30.284+0000 7fb3ecff1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:05:30.196 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:05:30.198 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:05:32.200 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T23:05:32.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:05:32.326+0000 7fad13a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:07:32.237 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:07:32.322 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:07:34.323 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T23:07:34.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:07:34.451+0000 7fc0c5d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:09:34.361 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:09:34.362 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:09:36.364 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T23:09:36.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:09:36.490+0000 7f4ad8eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:11:36.402 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:11:36.406 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:11:38.407 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T23:11:38.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:11:38.533+0000 7f377195d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:13:38.445 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:13:38.446 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:13:40.448 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T23:13:40.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:13:40.575+0000 7f06d0110640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:15:40.486 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:15:40.487 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:15:42.489 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T23:15:42.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:15:42.616+0000 7f5df966d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:17:42.527 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:17:42.528 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:17:44.529 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T23:17:44.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:17:44.656+0000 7fb6b1413640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:19:44.567 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:19:44.568 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:19:46.569 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T23:19:46.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:19:46.696+0000 7fedd366d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:21:46.607 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:21:46.608 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:21:48.610 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T23:21:48.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:21:48.737+0000 7f811d42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:23:48.647 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:23:48.648 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:23:50.649 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T23:23:50.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:23:50.777+0000 7fd029b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:25:50.687 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:25:50.688 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:25:52.690 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T23:25:52.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:25:52.816+0000 7f5b4de2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:27:52.728 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:27:52.729 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:27:54.730 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T23:27:54.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:27:54.857+0000 7f2ea27e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:29:54.768 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:29:54.769 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:29:56.770 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T23:29:56.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:29:56.898+0000 7ff40b4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:31:56.808 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:31:56.809 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:31:58.810 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T23:31:58.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:31:58.937+0000 7f7ef242d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:33:58.848 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:33:58.849 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:34:00.850 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T23:34:00.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:34:00.978+0000 7efc7db9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:36:00.888 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:36:00.889 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:36:02.890 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T23:36:03.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:36:03.018+0000 7f7de606d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:38:02.928 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:38:02.929 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:38:04.930 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T23:38:05.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:38:05.058+0000 7ff9290ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:40:04.968 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:40:04.969 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:40:06.970 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T23:40:07.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:40:07.100+0000 7f7bb00ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:42:07.007 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:42:07.008 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:42:09.010 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T23:42:09.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:42:09.137+0000 7fb6c946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:44:09.048 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:44:09.049 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:44:11.051 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T23:44:11.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:44:11.181+0000 7f231331d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:46:11.088 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:46:11.089 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:46:13.090 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T23:46:13.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:46:13.218+0000 7fb950d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:48:13.127 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:48:13.128 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:48:15.129 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T23:48:15.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:48:15.258+0000 7f63040ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:50:15.167 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:50:15.168 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:50:17.168 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T23:50:17.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:50:17.297+0000 7fdd4726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:52:17.207 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:52:17.208 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:52:19.209 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T23:52:19.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:52:19.337+0000 7fc9b8e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:54:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:54:19.249 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:54:21.251 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T23:54:21.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:54:21.378+0000 7f7fd3ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:56:21.290 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:56:21.291 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:56:23.293 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-01T23:56:23.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:56:23.418+0000 7fd978c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-01T23:58:23.331 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-01T23:58:23.332 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-01T23:58:25.334 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-01T23:58:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-01T23:58:25.458+0000 7f9a062ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:00:25.376 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:00:25.378 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:00:27.379 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T00:00:27.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:00:27.507+0000 7f713b55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:02:27.421 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:02:27.445 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:02:29.447 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T00:02:29.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:02:29.571+0000 7f467339d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:04:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:04:29.489 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:04:31.490 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T00:04:31.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:04:31.611+0000 7feea4ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:06:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:06:31.529 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:06:33.530 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T00:06:33.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:06:33.654+0000 7f926675d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:08:33.568 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:08:33.569 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:08:35.570 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T00:08:35.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:08:35.693+0000 7f98c275d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:10:35.609 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:10:35.610 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:10:37.612 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T00:10:37.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:10:37.734+0000 7faebb4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:12:37.651 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:12:37.652 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:12:39.653 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T00:12:39.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:12:39.774+0000 7f33e8e13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:14:39.692 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:14:39.693 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:14:41.693 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T00:14:41.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:14:41.814+0000 7ff56695d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:16:41.731 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:16:41.732 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:16:43.733 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T00:16:43.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:16:43.854+0000 7f575c26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:18:43.772 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:18:43.773 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:18:45.774 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T00:18:45.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:18:45.894+0000 7f3b8c06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:20:45.812 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:20:45.813 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:20:47.815 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T00:20:47.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:20:47.936+0000 7fe835d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:22:47.854 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:22:47.855 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:22:49.856 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T00:22:49.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:22:49.975+0000 7fad2c99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:24:49.894 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:24:49.895 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:24:51.897 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T00:24:52.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:24:52.016+0000 7f2238aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:26:51.936 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:26:51.937 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:26:53.938 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T00:26:54.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:26:54.057+0000 7f7ee066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:28:53.975 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:28:53.976 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:28:55.977 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T00:28:56.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:28:56.097+0000 7fd9a0024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:30:56.015 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:30:56.016 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:30:58.018 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T00:30:58.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:30:58.137+0000 7fce9df5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:32:58.056 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:32:58.057 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:33:00.059 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T00:33:00.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:33:00.179+0000 7f7059aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:35:00.097 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:35:00.098 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:35:02.099 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T00:35:02.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:35:02.219+0000 7f922ac6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:37:02.137 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:37:02.138 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:37:04.139 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T00:37:04.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:37:04.261+0000 7fc5bb55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:39:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:39:04.179 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:39:06.180 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T00:39:06.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:39:06.300+0000 7f564582d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:41:06.218 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:41:06.219 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:41:08.220 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T00:41:08.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:41:08.342+0000 7f1e0ed9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:43:08.259 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:43:08.260 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:43:10.261 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T00:43:10.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:43:10.384+0000 7fbc6386d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:45:10.299 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:45:10.300 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:45:12.301 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T00:45:12.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:45:12.421+0000 7f29d72ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:47:12.339 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:47:12.340 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:47:14.341 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T00:47:14.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:47:14.460+0000 7fc95b1d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:49:14.379 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:49:14.380 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:49:16.381 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T00:49:16.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:49:16.501+0000 7f829d42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:51:16.419 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:51:16.429 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:51:18.430 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T00:51:18.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:51:18.551+0000 7f30b742d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:53:18.468 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:53:18.469 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:53:20.470 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T00:53:20.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:53:20.591+0000 7f3fb7f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:55:20.508 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:55:20.509 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:55:22.510 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T00:55:22.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:55:22.630+0000 7fcd5a1d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:57:22.549 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:57:22.549 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:57:24.550 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T00:57:24.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:57:24.670+0000 7fac74213640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T00:59:24.588 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T00:59:24.589 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T00:59:26.590 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T00:59:26.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T00:59:26.710+0000 7fe2dee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T01:01:26.627 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T01:01:26.640 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T01:01:28.642 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T01:01:28.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T01:01:28.768+0000 7f90dbb1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T01:03:28.683 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T01:03:28.684 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T01:03:30.685 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T01:03:30.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T01:03:30.808+0000 7f997606d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T01:05:30.722 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T01:05:30.723 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T01:05:32.724 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T01:05:32.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T01:05:32.848+0000 7fabd959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T01:07:32.761 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T01:07:33.292 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T01:07:35.293 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T01:07:35.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T01:07:35.418+0000 7f85cc59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T01:09:35.330 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T01:09:35.331 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T01:09:37.332 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T01:09:37.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T01:09:37.457+0000 7f74490ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T01:11:37.369 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T01:11:37.370 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T01:11:39.371 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T01:11:39.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T01:11:39.498+0000 7fea8262d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T01:13:39.409 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T01:13:39.410 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T01:13:41.411 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T01:13:41.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T01:13:41.539+0000 7fb6c8114640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T01:15:41.449 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T01:15:41.450 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T01:15:43.451 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T01:15:43.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T01:15:43.580+0000 7f9f5de6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T01:17:43.489 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-02T01:17:43.490 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T01:17:45.491 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T01:17:45.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-02T01:17:45.619+0000 7f4d264ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T01:19:32.753 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-04-02T01:19:32.754 DEBUG:teuthology.task.console_log:Killing console logger for smithi059 2024-04-02T01:19:32.755 DEBUG:teuthology.task.console_log:Killing console logger for smithi087 2024-04-02T01:19:32.755 DEBUG:teuthology.exit:Finished running handlers