2024-04-16T07:29:00.125 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-16T07:29:00.126 DEBUG:teuthology.run:Teuthology command: teuthology --name matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi --archive /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657976 --verbose --owner scheduled_matan@teuthology --description crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} -- /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657976/orig.config.yaml 2024-04-16T07:29:00.166 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T07:29:00.249 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657976 branch: wip-matanb-crimson-replicated-april-15 description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} email: null first_in_suite: false job_id: '7657976' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-replicated-april-15 ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: min_in: 4 workunit: branch: wip-matanb-crimson-replicated-april-15 env: CRIMSON_COMPAT: '1' sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 7594 sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-matanb-crimson-replicated-april-15 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb targets: smithi070.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMsJ2TJdaZN5y4l3u/X2+t7XwFtINmipTx2SWpFb/+DHE7lIYCT/PxtPsat5G3qtz/BI6Ro1PNqxidKlyqL4WcI= smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMWPjIiC9oiD/C0llv5iLJ6LthJN1mBh2pA/Bpd0Ecmd+PM/Bw6gew0/xXSdD3v7vbwk6YRVobobOscRqkt5Ums= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson - thrashosds: ceph_objectstore_tool: false chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - rados: clients: - client.0 objects: 500 op_weights: write: 100 ops: 4000 write_fadvise_dontneed: true teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-16_07:22:04 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-16T07:29:00.249 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa; will attempt to use it 2024-04-16T07:29:00.250 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks 2024-04-16T07:29:00.250 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-16T07:29:00.252 INFO:teuthology.task.internal:Checking packages... 2024-04-16T07:29:00.275 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '30b0aa0009a925d6a46536e55c5656edcd51ecdb' 2024-04-16T07:29:00.275 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T07:29:00.275 INFO:teuthology.packaging:ref: None 2024-04-16T07:29:00.275 INFO:teuthology.packaging:tag: None 2024-04-16T07:29:00.275 INFO:teuthology.packaging:branch: wip-matanb-crimson-replicated-april-15 2024-04-16T07:29:00.275 INFO:teuthology.packaging:sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:29:00.276 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-replicated-april-15 2024-04-16T07:29:00.417 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3109.g30b0aa00 2024-04-16T07:29:00.419 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-16T07:29:00.427 INFO:teuthology.task.internal:no buildpackages task found 2024-04-16T07:29:00.427 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-16T07:29:00.445 INFO:teuthology.task.internal:Saving configuration 2024-04-16T07:29:00.459 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-16T07:29:00.470 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T07:29:00.495 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657976', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 07:23:13.249089', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMsJ2TJdaZN5y4l3u/X2+t7XwFtINmipTx2SWpFb/+DHE7lIYCT/PxtPsat5G3qtz/BI6Ro1PNqxidKlyqL4WcI='} 2024-04-16T07:29:00.516 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657976', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 07:23:13.250377', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMWPjIiC9oiD/C0llv5iLJ6LthJN1mBh2pA/Bpd0Ecmd+PM/Bw6gew0/xXSdD3v7vbwk6YRVobobOscRqkt5Ums='} 2024-04-16T07:29:00.516 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-16T07:29:00.527 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-04-16T07:29:00.527 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-04-16T07:29:00.527 INFO:teuthology.run_tasks:Running task console_log... 2024-04-16T07:29:00.604 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f371b2ade50>, signals=[15]) 2024-04-16T07:29:00.605 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-16T07:29:00.616 INFO:teuthology.task.internal:Opening connections... 2024-04-16T07:29:00.616 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi070.front.sepia.ceph.com 2024-04-16T07:29:00.618 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:29:00.727 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2024-04-16T07:29:00.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:29:00.833 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-16T07:29:00.842 DEBUG:teuthology.orchestra.run.smithi070:> uname -m 2024-04-16T07:29:00.859 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2024-04-16T07:29:00.859 DEBUG:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2024-04-16T07:29:00.916 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="CentOS Stream" 2024-04-16T07:29:00.916 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="9" 2024-04-16T07:29:00.916 INFO:teuthology.orchestra.run.smithi070.stdout:ID="centos" 2024-04-16T07:29:00.916 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="rhel fedora" 2024-04-16T07:29:00.916 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="9" 2024-04-16T07:29:00.916 INFO:teuthology.orchestra.run.smithi070.stdout:PLATFORM_ID="platform:el9" 2024-04-16T07:29:00.916 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T07:29:00.916 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31" 2024-04-16T07:29:00.916 INFO:teuthology.orchestra.run.smithi070.stdout:LOGO="fedora-logo-icon" 2024-04-16T07:29:00.917 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T07:29:00.917 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://centos.org/" 2024-04-16T07:29:00.917 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T07:29:00.917 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T07:29:00.917 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T07:29:00.917 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2024-04-16T07:29:00.939 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2024-04-16T07:29:00.957 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2024-04-16T07:29:00.957 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2024-04-16T07:29:01.014 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="CentOS Stream" 2024-04-16T07:29:01.014 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="9" 2024-04-16T07:29:01.015 INFO:teuthology.orchestra.run.smithi172.stdout:ID="centos" 2024-04-16T07:29:01.015 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="rhel fedora" 2024-04-16T07:29:01.015 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="9" 2024-04-16T07:29:01.015 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el9" 2024-04-16T07:29:01.015 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T07:29:01.015 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2024-04-16T07:29:01.016 INFO:teuthology.orchestra.run.smithi172.stdout:LOGO="fedora-logo-icon" 2024-04-16T07:29:01.016 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T07:29:01.016 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://centos.org/" 2024-04-16T07:29:01.016 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T07:29:01.016 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T07:29:01.016 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T07:29:01.017 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2024-04-16T07:29:01.040 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-16T07:29:01.054 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-16T07:29:01.064 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-16T07:29:01.065 DEBUG:teuthology.orchestra.run.smithi070:> test '!' -e /home/ubuntu/cephtest 2024-04-16T07:29:01.068 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2024-04-16T07:29:01.085 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-16T07:29:01.096 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-16T07:29:01.097 DEBUG:teuthology.orchestra.run.smithi070:> test -z $(ls -A /var/lib/ceph) 2024-04-16T07:29:01.125 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2024-04-16T07:29:01.225 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-16T07:29:01.277 INFO:teuthology.run_tasks:Running task kernel... 2024-04-16T07:29:01.296 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-16T07:29:01.296 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-16T07:29:01.297 DEBUG:teuthology.orchestra.run.smithi070:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T07:29:01.297 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T07:29:01.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:29:01.313 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-04-16T07:29:01.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:29:01.314 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-04-16T07:29:01.370 INFO:teuthology.orchestra.run.smithi070.stdout:5.14.0-361.el9.x86_64 2024-04-16T07:29:01.370 INFO:teuthology.task.kernel:Running kernel on smithi070: 5.14.0-361.el9.x86_64 2024-04-16T07:29:01.370 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum install -y kernel 2024-04-16T07:29:01.372 INFO:teuthology.orchestra.run.smithi172.stdout:5.14.0-361.el9.x86_64 2024-04-16T07:29:01.372 INFO:teuthology.task.kernel:Running kernel on smithi172: 5.14.0-361.el9.x86_64 2024-04-16T07:29:01.372 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum install -y kernel 2024-04-16T07:29:03.502 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-04-16T07:29:03.570 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-04-16T07:29:05.210 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS-9 - BaseOS 3.3 MB/s | 2.0 MB 00:00 2024-04-16T07:29:05.495 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - BaseOS 2.5 MB/s | 2.0 MB 00:00 2024-04-16T07:29:10.044 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - BaseOS 1.9 MB/s | 8.0 MB 00:04 2024-04-16T07:29:15.390 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 836 kB/s | 8.0 MB 00:09 2024-04-16T07:29:15.936 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - AppStream 4.0 MB/s | 19 MB 00:04 2024-04-16T07:29:21.244 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.2 MB 00:01 2024-04-16T07:29:21.440 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 3.8 MB/s | 19 MB 00:04 2024-04-16T07:29:22.820 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-04-16T07:29:24.038 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-04-16T07:29:26.175 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 6.2 MB 00:01 2024-04-16T07:29:27.816 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-16T07:29:29.055 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-04-16T07:29:30.143 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 9.5 MB/s | 21 MB 00:02 2024-04-16T07:29:31.461 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-16T07:29:31.462 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-16T07:29:31.547 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:29:31.555 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:29:31.555 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-16T07:29:31.555 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:29:31.555 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-16T07:29:31.555 INFO:teuthology.orchestra.run.smithi172.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-16T07:29:31.555 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-16T07:29:31.555 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-16T07:29:31.555 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-16T07:29:31.555 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-16T07:29:31.555 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:29:31.556 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-16T07:29:31.556 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:29:31.556 INFO:teuthology.orchestra.run.smithi172.stdout:Install 4 Packages 2024-04-16T07:29:31.556 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:29:31.556 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 77 M 2024-04-16T07:29:31.556 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 123 M 2024-04-16T07:29:31.556 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-16T07:29:31.929 INFO:teuthology.orchestra.run.smithi172.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 745 kB/s | 277 kB 00:00 2024-04-16T07:29:32.831 INFO:teuthology.orchestra.run.smithi172.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 12 MB/s | 15 MB 00:01 2024-04-16T07:29:34.073 INFO:teuthology.orchestra.run.smithi172.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 13 MB/s | 34 MB 00:02 2024-04-16T07:29:34.598 INFO:teuthology.orchestra.run.smithi172.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 10 MB/s | 28 MB 00:02 2024-04-16T07:29:34.601 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:29:34.601 INFO:teuthology.orchestra.run.smithi172.stdout:Total 25 MB/s | 77 MB 00:03 2024-04-16T07:29:34.602 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-16T07:29:34.969 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-16T07:29:34.969 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-16T07:29:35.242 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-16T07:29:35.803 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-16T07:29:35.803 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-16T07:29:37.624 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-16T07:29:37.624 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-16T07:29:37.729 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:29:37.734 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:29:37.735 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-04-16T07:29:37.735 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:29:37.735 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-16T07:29:37.735 INFO:teuthology.orchestra.run.smithi070.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-16T07:29:37.735 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-04-16T07:29:37.735 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-16T07:29:37.735 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-16T07:29:37.735 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-16T07:29:37.735 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:29:37.735 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-16T07:29:37.735 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:29:37.735 INFO:teuthology.orchestra.run.smithi070.stdout:Install 4 Packages 2024-04-16T07:29:37.735 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:29:37.736 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 77 M 2024-04-16T07:29:37.736 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 123 M 2024-04-16T07:29:37.736 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-16T07:29:37.893 INFO:teuthology.orchestra.run.smithi070.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.7 MB/s | 277 kB 00:00 2024-04-16T07:29:38.120 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-16T07:29:38.477 INFO:teuthology.orchestra.run.smithi070.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-04-16T07:29:38.710 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-16T07:29:38.790 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:29:39.027 INFO:teuthology.orchestra.run.smithi070.stdout:(3/4): kernel-modules-core-5.14.0-438.el9.x86_6 24 MB/s | 28 MB 00:01 2024-04-16T07:29:39.402 INFO:teuthology.orchestra.run.smithi070.stdout:(4/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 20 MB/s | 34 MB 00:01 2024-04-16T07:29:39.405 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:29:39.405 INFO:teuthology.orchestra.run.smithi070.stdout:Total 46 MB/s | 77 MB 00:01 2024-04-16T07:29:39.405 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-16T07:29:39.775 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-16T07:29:39.775 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-16T07:29:40.610 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-16T07:29:40.610 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-16T07:29:41.549 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:29:41.694 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:29:42.913 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-16T07:29:43.538 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-16T07:29:43.635 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:29:46.410 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:29:46.415 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:29:46.554 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:29:46.610 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:29:51.247 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:29:51.346 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:29:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:29:56.056 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:20.544 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:20.545 INFO:teuthology.orchestra.run.smithi172.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T07:30:20.545 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:30:20.588 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:21.001 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:21.001 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-16T07:30:21.001 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:30:21.001 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:30:21.422 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:21.423 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:30:21.423 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-16T07:30:21.423 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-16T07:30:21.423 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-16T07:30:21.423 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-16T07:30:21.423 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-16T07:30:21.423 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:30:21.423 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:30:21.788 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T07:30:21.887 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T07:30:21.887 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T07:30:21.887 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-16T07:30:21.888 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-16T07:30:21.888 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-16T07:30:21.888 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-04-16T07:30:21.888 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2024-04-16T07:30:21.888 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T07:30:21.968 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T07:30:21.968 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T07:30:21.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo rpm -qi grub2-tools 2024-04-16T07:30:22.002 INFO:teuthology.orchestra.run.smithi172.stdout:Name : grub2-tools 2024-04-16T07:30:22.002 INFO:teuthology.orchestra.run.smithi172.stdout:Epoch : 1 2024-04-16T07:30:22.002 INFO:teuthology.orchestra.run.smithi172.stdout:Version : 2.06 2024-04-16T07:30:22.002 INFO:teuthology.orchestra.run.smithi172.stdout:Release : 68.el9 2024-04-16T07:30:22.003 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: x86_64 2024-04-16T07:30:22.003 INFO:teuthology.orchestra.run.smithi172.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T07:30:22.003 INFO:teuthology.orchestra.run.smithi172.stdout:Group : Unspecified 2024-04-16T07:30:22.003 INFO:teuthology.orchestra.run.smithi172.stdout:Size : 8264254 2024-04-16T07:30:22.003 INFO:teuthology.orchestra.run.smithi172.stdout:License : GPLv3+ 2024-04-16T07:30:22.003 INFO:teuthology.orchestra.run.smithi172.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T07:30:22.003 INFO:teuthology.orchestra.run.smithi172.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T07:30:22.003 INFO:teuthology.orchestra.run.smithi172.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T07:30:22.003 INFO:teuthology.orchestra.run.smithi172.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T07:30:22.003 INFO:teuthology.orchestra.run.smithi172.stdout:Packager : builder@centos.org 2024-04-16T07:30:22.003 INFO:teuthology.orchestra.run.smithi172.stdout:Vendor : CentOS 2024-04-16T07:30:22.003 INFO:teuthology.orchestra.run.smithi172.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T07:30:22.003 INFO:teuthology.orchestra.run.smithi172.stdout:Summary : Support tools for GRUB. 2024-04-16T07:30:22.003 INFO:teuthology.orchestra.run.smithi172.stdout:Description : 2024-04-16T07:30:22.003 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:30:22.003 INFO:teuthology.orchestra.run.smithi172.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T07:30:22.004 INFO:teuthology.orchestra.run.smithi172.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T07:30:22.004 INFO:teuthology.orchestra.run.smithi172.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T07:30:22.004 INFO:teuthology.orchestra.run.smithi172.stdout:hardware devices. 2024-04-16T07:30:22.004 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:30:22.004 INFO:teuthology.orchestra.run.smithi172.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T07:30:22.004 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T07:30:22.004 INFO:teuthology.task.kernel:Updating grub on smithi172 to boot 5.14.0-438.el9.x86_64 2024-04-16T07:30:22.005 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T07:30:22.591 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2024-04-16T07:30:23.930 INFO:teuthology.orchestra.run.smithi172.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T07:30:23.967 INFO:teuthology.orchestra.run.smithi172.stderr:done 2024-04-16T07:30:23.970 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-04-16T07:30:23.985 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.TuyxObMwtE 2024-04-16T07:30:23.985 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TuyxObMwtE 2024-04-16T07:30:24.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.TuyxObMwtE 2024-04-16T07:30:24.186 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.TuyxObMwtE is 6KB 2024-04-16T07:30:24.235 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.TuyxObMwtE 2024-04-16T07:30:24.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T07:30:24.314 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T07:30:24.314 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T07:30:24.314 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T07:30:24.314 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-16T07:30:24.315 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-16T07:30:24.451 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2024-04-16T07:30:25.374 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:25.374 INFO:teuthology.orchestra.run.smithi070.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T07:30:25.374 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:30:25.418 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:25.827 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:25.827 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-16T07:30:25.827 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:30:25.827 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:30:26.236 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:26.236 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:30:26.236 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-16T07:30:26.237 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-16T07:30:26.237 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-16T07:30:26.237 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-16T07:30:26.237 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-16T07:30:26.237 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:30:26.237 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:30:26.578 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T07:30:26.728 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T07:30:26.729 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T07:30:26.729 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-16T07:30:26.729 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-16T07:30:26.729 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-16T07:30:26.729 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-16T07:30:26.729 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi070.front.sepia.ceph.com, path=None, version=distro) 2024-04-16T07:30:26.730 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T07:30:26.810 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T07:30:26.810 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T07:30:26.811 DEBUG:teuthology.orchestra.run.smithi070:> sudo rpm -qi grub2-tools 2024-04-16T07:30:26.853 INFO:teuthology.orchestra.run.smithi070.stdout:Name : grub2-tools 2024-04-16T07:30:26.854 INFO:teuthology.orchestra.run.smithi070.stdout:Epoch : 1 2024-04-16T07:30:26.854 INFO:teuthology.orchestra.run.smithi070.stdout:Version : 2.06 2024-04-16T07:30:26.854 INFO:teuthology.orchestra.run.smithi070.stdout:Release : 68.el9 2024-04-16T07:30:26.854 INFO:teuthology.orchestra.run.smithi070.stdout:Architecture: x86_64 2024-04-16T07:30:26.854 INFO:teuthology.orchestra.run.smithi070.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T07:30:26.854 INFO:teuthology.orchestra.run.smithi070.stdout:Group : Unspecified 2024-04-16T07:30:26.854 INFO:teuthology.orchestra.run.smithi070.stdout:Size : 8264254 2024-04-16T07:30:26.854 INFO:teuthology.orchestra.run.smithi070.stdout:License : GPLv3+ 2024-04-16T07:30:26.854 INFO:teuthology.orchestra.run.smithi070.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T07:30:26.854 INFO:teuthology.orchestra.run.smithi070.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T07:30:26.854 INFO:teuthology.orchestra.run.smithi070.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T07:30:26.854 INFO:teuthology.orchestra.run.smithi070.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T07:30:26.854 INFO:teuthology.orchestra.run.smithi070.stdout:Packager : builder@centos.org 2024-04-16T07:30:26.854 INFO:teuthology.orchestra.run.smithi070.stdout:Vendor : CentOS 2024-04-16T07:30:26.854 INFO:teuthology.orchestra.run.smithi070.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T07:30:26.855 INFO:teuthology.orchestra.run.smithi070.stdout:Summary : Support tools for GRUB. 2024-04-16T07:30:26.855 INFO:teuthology.orchestra.run.smithi070.stdout:Description : 2024-04-16T07:30:26.855 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:30:26.855 INFO:teuthology.orchestra.run.smithi070.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T07:30:26.855 INFO:teuthology.orchestra.run.smithi070.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T07:30:26.855 INFO:teuthology.orchestra.run.smithi070.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T07:30:26.855 INFO:teuthology.orchestra.run.smithi070.stdout:hardware devices. 2024-04-16T07:30:26.855 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:30:26.855 INFO:teuthology.orchestra.run.smithi070.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T07:30:26.856 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T07:30:26.856 INFO:teuthology.task.kernel:Updating grub on smithi070 to boot 5.14.0-438.el9.x86_64 2024-04-16T07:30:26.856 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T07:30:27.472 INFO:teuthology.orchestra.run.smithi070.stderr:Generating grub configuration file ... 2024-04-16T07:30:28.930 INFO:teuthology.orchestra.run.smithi070.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T07:30:28.949 INFO:teuthology.orchestra.run.smithi070.stderr:done 2024-04-16T07:30:28.952 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-04-16T07:30:28.967 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.aO5pQQctiC 2024-04-16T07:30:28.967 DEBUG:teuthology.orchestra.run.smithi070:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.aO5pQQctiC 2024-04-16T07:30:29.042 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.aO5pQQctiC 2024-04-16T07:30:29.186 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.aO5pQQctiC is 6KB 2024-04-16T07:30:29.234 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.aO5pQQctiC 2024-04-16T07:30:29.248 DEBUG:teuthology.orchestra.run.smithi070:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T07:30:29.313 INFO:teuthology.orchestra.run.smithi070.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T07:30:29.313 INFO:teuthology.orchestra.run.smithi070.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T07:30:29.313 INFO:teuthology.orchestra.run.smithi070.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T07:30:29.313 INFO:teuthology.orchestra.run.smithi070.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-16T07:30:29.315 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-16T07:30:29.458 DEBUG:teuthology.orchestra.run.smithi070:> sudo shutdown -r now 2024-04-16T07:30:54.478 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-04-16T07:30:54.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-04-16T07:30:54.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:30:59.465 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-16T07:30:59.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-16T07:30:59.466 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:31:12.802 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-04-16T07:31:17.918 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.70 2024-04-16T07:31:21.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-04-16T07:31:21.808 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:31:24.862 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-04-16T07:31:26.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-16T07:31:26.923 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:31:36.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-04-16T07:31:36.870 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:32:26.975 DEBUG:teuthology.orchestra.remote:timed out 2024-04-16T07:32:36.877 DEBUG:teuthology.orchestra.remote:timed out 2024-04-16T07:32:38.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-16T07:32:38.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:32:39.399 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-04-16T07:32:39.774 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-16T07:32:39.774 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-438.el9.x86_64"... 2024-04-16T07:32:39.775 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-04-16T07:32:39.791 INFO:teuthology.orchestra.run.smithi070.stdout:5.14.0-438.el9.x86_64 2024-04-16T07:32:39.792 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-16T07:32:39.792 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T07:32:39.792 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T07:32:40.793 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-16T07:32:40.793 DEBUG:teuthology.orchestra.run.smithi070:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T07:32:40.866 INFO:teuthology.orchestra.run.smithi070.stdout:ttyS1 2024-04-16T07:32:40.906 DEBUG:teuthology.parallel:result is None 2024-04-16T07:32:51.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-04-16T07:32:51.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:32:52.203 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-04-16T07:32:52.591 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-04-16T07:32:52.591 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-438.el9.x86_64"... 2024-04-16T07:32:52.592 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-04-16T07:32:52.607 INFO:teuthology.orchestra.run.smithi172.stdout:5.14.0-438.el9.x86_64 2024-04-16T07:32:52.607 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-16T07:32:52.608 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T07:32:52.608 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T07:32:53.610 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-04-16T07:32:53.610 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T07:32:53.697 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2024-04-16T07:32:53.730 DEBUG:teuthology.parallel:result is None 2024-04-16T07:32:53.731 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-16T07:32:53.739 INFO:teuthology.task.internal:Creating test directory... 2024-04-16T07:32:53.739 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T07:32:53.742 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T07:32:53.760 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-16T07:32:53.771 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-16T07:32:53.784 INFO:teuthology.task.internal:Creating archive directory... 2024-04-16T07:32:53.784 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T07:32:53.801 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T07:32:53.856 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-16T07:32:53.868 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-16T07:32:53.868 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T07:32:53.870 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T07:32:53.912 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:32:53.925 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:32:53.932 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:32:53.947 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:32:53.949 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-16T07:32:53.956 INFO:teuthology.task.internal:Configuring sudo... 2024-04-16T07:32:53.956 DEBUG:teuthology.orchestra.run.smithi070:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T07:32:53.962 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T07:32:53.998 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-16T07:32:54.009 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-16T07:32:54.009 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T07:32:54.033 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T07:32:54.054 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:32:54.132 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:32:54.185 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:32:54.269 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:32:54.314 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-16T07:32:54.314 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T07:32:54.383 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:32:54.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:32:54.497 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:32:54.539 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:32:54.586 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-16T07:32:54.586 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T07:32:54.656 DEBUG:teuthology.orchestra.run.smithi070:> sudo service rsyslog restart 2024-04-16T07:32:54.659 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2024-04-16T07:32:54.701 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T07:32:54.756 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T07:32:55.121 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-16T07:32:55.129 INFO:teuthology.task.internal:Starting timer... 2024-04-16T07:32:55.129 INFO:teuthology.run_tasks:Running task pcp... 2024-04-16T07:32:55.144 INFO:teuthology.run_tasks:Running task selinux... 2024-04-16T07:32:55.153 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-16T07:32:55.154 DEBUG:teuthology.orchestra.run.smithi070:> sudo service auditd rotate 2024-04-16T07:32:55.221 INFO:teuthology.orchestra.run.smithi070.stdout:Rotating logs: 2024-04-16T07:32:55.222 DEBUG:teuthology.orchestra.run.smithi172:> sudo service auditd rotate 2024-04-16T07:32:55.287 INFO:teuthology.orchestra.run.smithi172.stdout:Rotating logs: 2024-04-16T07:32:55.289 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-16T07:32:55.289 DEBUG:teuthology.orchestra.run.smithi070:> /usr/sbin/getenforce 2024-04-16T07:32:55.310 INFO:teuthology.orchestra.run.smithi070.stdout:Permissive 2024-04-16T07:32:55.310 DEBUG:teuthology.orchestra.run.smithi172:> /usr/sbin/getenforce 2024-04-16T07:32:55.335 INFO:teuthology.orchestra.run.smithi172.stdout:Permissive 2024-04-16T07:32:55.336 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi070.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi172.front.sepia.ceph.com': 'permissive'} 2024-04-16T07:32:55.336 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T07:32:55.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:32:55.379 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T07:32:55.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:32:55.409 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-16T07:32:55.409 DEBUG:teuthology.orchestra.run.smithi070:> sudo /usr/sbin/setenforce permissive 2024-04-16T07:32:55.462 DEBUG:teuthology.orchestra.run.smithi172:> sudo /usr/sbin/setenforce permissive 2024-04-16T07:32:55.496 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-16T07:32:55.508 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-16T07:32:55.527 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-16T07:32:55.528 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi070.front.sepia.ceph.com,smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-16T07:37:16.186 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi070.front.sepia.ceph.com'), Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2024-04-16T07:37:16.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-16T07:37:16.189 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:37:16.298 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-04-16T07:37:16.379 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-16T07:37:16.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-04-16T07:37:16.380 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:37:16.527 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-04-16T07:37:16.607 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-04-16T07:37:16.607 INFO:teuthology.run_tasks:Running task clock... 2024-04-16T07:37:16.618 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-16T07:37:16.619 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T07:37:16.620 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T07:37:16.624 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T07:37:16.624 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T07:37:16.656 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T07:37:16.672 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T07:37:16.680 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T07:37:16.698 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T07:37:16.702 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: ntpd: command not found 2024-04-16T07:37:16.717 INFO:teuthology.orchestra.run.smithi070.stdout:506 Cannot talk to daemon 2024-04-16T07:37:16.729 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: ntpd: command not found 2024-04-16T07:37:16.734 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T07:37:16.743 INFO:teuthology.orchestra.run.smithi172.stdout:506 Cannot talk to daemon 2024-04-16T07:37:16.751 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T07:37:16.761 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T07:37:16.778 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T07:37:16.805 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-04-16T07:37:16.809 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T07:37:16.809 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-04-16T07:37:16.809 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:37:16.809 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:37:16.809 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:37:16.809 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:37:16.831 INFO:teuthology.orchestra.run.smithi172.stderr:bash: line 1: ntpq: command not found 2024-04-16T07:37:16.835 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T07:37:16.835 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2024-04-16T07:37:16.835 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:37:16.835 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:37:16.835 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:37:16.835 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:37:16.835 INFO:teuthology.run_tasks:Running task install... 2024-04-16T07:37:16.845 DEBUG:teuthology.task.install:project ceph 2024-04-16T07:37:16.845 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}} 2024-04-16T07:37:16.846 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'} 2024-04-16T07:37:16.846 INFO:teuthology.task.install:Using flavor: crimson 2024-04-16T07:37:16.853 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-16T07:37:16.854 INFO:teuthology.task.install:extra packages: [] 2024-04-16T07:37:16.855 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'tag': None, 'wait_for_package': False} 2024-04-16T07:37:16.856 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:37:16.859 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'tag': None, 'wait_for_package': False} 2024-04-16T07:37:16.860 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:37:17.015 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/ 2024-04-16T07:37:17.015 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3109.g30b0aa00 2024-04-16T07:37:17.019 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/ 2024-04-16T07:37:17.019 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3109.g30b0aa00 2024-04-16T07:37:17.300 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T07:37:17.301 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-16T07:37:17.301 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T07:37:17.331 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T07:37:17.331 DEBUG:teuthology.orchestra.run.smithi070:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/30b0aa0009a925d6a46536e55c5656edcd51ecdb/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T07:37:17.357 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T07:37:17.358 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-16T07:37:17.358 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T07:37:17.389 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T07:37:17.389 DEBUG:teuthology.orchestra.run.smithi172:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/30b0aa0009a925d6a46536e55c5656edcd51ecdb/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T07:37:17.400 DEBUG:teuthology.orchestra.run.smithi070:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T07:37:17.459 DEBUG:teuthology.orchestra.run.smithi172:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T07:37:17.483 DEBUG:teuthology.orchestra.run.smithi070:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T07:37:17.543 DEBUG:teuthology.orchestra.run.smithi172:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T07:37:17.552 INFO:teuthology.orchestra.run.smithi070.stdout:check_obsoletes = 1 2024-04-16T07:37:17.554 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum clean all 2024-04-16T07:37:17.611 INFO:teuthology.orchestra.run.smithi172.stdout:check_obsoletes = 1 2024-04-16T07:37:17.612 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum clean all 2024-04-16T07:37:17.803 INFO:teuthology.orchestra.run.smithi070.stdout:74 files removed 2024-04-16T07:37:17.829 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-radosgw 2024-04-16T07:37:17.869 INFO:teuthology.orchestra.run.smithi172.stdout:74 files removed 2024-04-16T07:37:17.901 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-radosgw 2024-04-16T07:37:18.793 INFO:teuthology.orchestra.run.smithi070.stdout:ceph packages for x86_64 132 kB/s | 87 kB 00:00 2024-04-16T07:37:19.004 INFO:teuthology.orchestra.run.smithi172.stdout:ceph packages for x86_64 109 kB/s | 87 kB 00:00 2024-04-16T07:37:19.332 INFO:teuthology.orchestra.run.smithi070.stdout:ceph noarch packages 24 kB/s | 12 kB 00:00 2024-04-16T07:37:19.805 INFO:teuthology.orchestra.run.smithi172.stdout:ceph noarch packages 15 kB/s | 12 kB 00:00 2024-04-16T07:37:20.083 INFO:teuthology.orchestra.run.smithi070.stdout:ceph source packages 2.9 kB/s | 2.0 kB 00:00 2024-04-16T07:37:20.489 INFO:teuthology.orchestra.run.smithi172.stdout:ceph source packages 3.2 kB/s | 2.0 kB 00:00 2024-04-16T07:37:20.718 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-04-16T07:37:21.332 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - AppStream 8.3 MB/s | 6.7 MB 00:00 2024-04-16T07:37:22.102 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-04-16T07:37:22.850 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2024-04-16T07:37:23.917 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.0 MB 00:01 2024-04-16T07:37:24.284 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - BaseOS 7.2 MB/s | 8.0 MB 00:01 2024-04-16T07:37:26.160 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-04-16T07:37:30.564 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 6.2 MB 00:00 2024-04-16T07:37:32.980 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-04-16T07:37:34.833 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-16T07:37:39.889 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-04-16T07:37:42.642 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:37:42.659 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:37:42.659 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-04-16T07:37:42.659 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:37:42.659 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-16T07:37:42.660 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 10 M 2024-04-16T07:37:42.660 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrading: 2024-04-16T07:37:42.660 INFO:teuthology.orchestra.run.smithi070.stdout: librados2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.4 M 2024-04-16T07:37:42.660 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.7 M 2024-04-16T07:37:42.660 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-04-16T07:37:42.660 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options 2024-04-16T07:37:42.660 INFO:teuthology.orchestra.run.smithi070.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T07:37:42.660 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.9 M 2024-04-16T07:37:42.660 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 19 M 2024-04-16T07:37:42.660 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 25 k 2024-04-16T07:37:42.660 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T07:37:42.660 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T07:37:42.660 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T07:37:42.660 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 797 k 2024-04-16T07:37:42.661 INFO:teuthology.orchestra.run.smithi070.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T07:37:42.661 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T07:37:42.661 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T07:37:42.661 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 546 k 2024-04-16T07:37:42.661 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T07:37:42.661 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.2 M 2024-04-16T07:37:42.661 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T07:37:42.661 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T07:37:42.661 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T07:37:42.661 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse 2024-04-16T07:37:42.661 INFO:teuthology.orchestra.run.smithi070.stdout: x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 45 k 2024-04-16T07:37:42.661 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 135 k 2024-04-16T07:37:42.662 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 159 k 2024-04-16T07:37:42.662 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T07:37:42.662 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 315 k 2024-04-16T07:37:42.662 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 293 k 2024-04-16T07:37:42.662 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 99 k 2024-04-16T07:37:42.662 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-16T07:37:42.662 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-16T07:37:42.662 INFO:teuthology.orchestra.run.smithi070.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T07:37:42.662 INFO:teuthology.orchestra.run.smithi070.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T07:37:42.662 INFO:teuthology.orchestra.run.smithi070.stdout:Installing weak dependencies: 2024-04-16T07:37:42.662 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-16T07:37:42.662 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:37:42.662 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-16T07:37:42.662 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:37:42.663 INFO:teuthology.orchestra.run.smithi070.stdout:Install 30 Packages 2024-04-16T07:37:42.663 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrade 2 Packages 2024-04-16T07:37:42.663 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:37:42.664 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 59 M 2024-04-16T07:37:42.664 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-16T07:37:42.697 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 19 MB 00:17 2024-04-16T07:37:47.464 INFO:teuthology.orchestra.run.smithi070.stdout:(1/32): ceph-base-19.0.0-3109.g30b0aa00.el9.x86 1.3 MB/s | 5.9 MB 00:04 2024-04-16T07:37:47.589 INFO:teuthology.orchestra.run.smithi070.stdout:(2/32): ceph-selinux-19.0.0-3109.g30b0aa00.el9. 202 kB/s | 25 kB 00:00 2024-04-16T07:37:47.898 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.2 MB 00:01 2024-04-16T07:37:48.922 INFO:teuthology.orchestra.run.smithi070.stdout:(3/32): libcephfs2-19.0.0-3109.g30b0aa00.el9.x8 598 kB/s | 797 kB 00:01 2024-04-16T07:37:49.532 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-16T07:37:49.597 INFO:teuthology.orchestra.run.smithi070.stdout:(4/32): libradosstriper1-19.0.0-3109.g30b0aa00. 810 kB/s | 546 kB 00:00 2024-04-16T07:37:51.634 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-16T07:37:52.039 INFO:teuthology.orchestra.run.smithi070.stdout:(5/32): librgw2-19.0.0-3109.g30b0aa00.el9.x86_6 2.1 MB/s | 5.2 MB 00:02 2024-04-16T07:37:52.156 INFO:teuthology.orchestra.run.smithi070.stdout:(6/32): python3-ceph-argparse-19.0.0-3109.g30b0 387 kB/s | 45 kB 00:00 2024-04-16T07:37:52.273 INFO:teuthology.orchestra.run.smithi070.stdout:(7/32): python3-ceph-common-19.0.0-3109.g30b0aa 1.1 MB/s | 135 kB 00:00 2024-04-16T07:37:52.690 INFO:teuthology.orchestra.run.smithi070.stdout:(8/32): python3-cephfs-19.0.0-3109.g30b0aa00.el 383 kB/s | 159 kB 00:00 2024-04-16T07:37:52.948 INFO:teuthology.orchestra.run.smithi070.stdout:(9/32): python3-rados-19.0.0-3109.g30b0aa00.el9 1.2 MB/s | 315 kB 00:00 2024-04-16T07:37:53.148 INFO:teuthology.orchestra.run.smithi070.stdout:(10/32): python3-rbd-19.0.0-3109.g30b0aa00.el9. 1.4 MB/s | 293 kB 00:00 2024-04-16T07:37:53.265 INFO:teuthology.orchestra.run.smithi070.stdout:(11/32): python3-rgw-19.0.0-3109.g30b0aa00.el9. 853 kB/s | 99 kB 00:00 2024-04-16T07:37:53.449 INFO:teuthology.orchestra.run.smithi070.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 576 kB/s | 106 kB 00:00 2024-04-16T07:37:53.500 INFO:teuthology.orchestra.run.smithi070.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-04-16T07:37:53.533 INFO:teuthology.orchestra.run.smithi070.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-04-16T07:37:53.600 INFO:teuthology.orchestra.run.smithi070.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-04-16T07:37:53.642 INFO:teuthology.orchestra.run.smithi070.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-04-16T07:37:53.684 INFO:teuthology.orchestra.run.smithi070.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 988 kB/s | 41 kB 00:00 2024-04-16T07:37:53.726 INFO:teuthology.orchestra.run.smithi070.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2024-04-16T07:37:53.761 INFO:teuthology.orchestra.run.smithi070.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 35 kB 00:00 2024-04-16T07:37:53.793 INFO:teuthology.orchestra.run.smithi070.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-04-16T07:37:54.060 INFO:teuthology.orchestra.run.smithi070.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-04-16T07:37:54.302 INFO:teuthology.orchestra.run.smithi070.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-04-16T07:37:54.360 INFO:teuthology.orchestra.run.smithi070.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-04-16T07:37:54.419 INFO:teuthology.orchestra.run.smithi070.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2024-04-16T07:37:54.478 INFO:teuthology.orchestra.run.smithi070.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-16T07:37:54.586 INFO:teuthology.orchestra.run.smithi070.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 838 kB 00:00 2024-04-16T07:37:54.653 INFO:teuthology.orchestra.run.smithi070.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-04-16T07:37:54.754 INFO:teuthology.orchestra.run.smithi070.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-16T07:37:56.187 INFO:teuthology.orchestra.run.smithi070.stdout:(29/32): librados2-19.0.0-3109.g30b0aa00.el9.x8 2.4 MB/s | 3.4 MB 00:01 2024-04-16T07:37:56.875 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 5.3 kB/s | 1.7 kB 00:00 2024-04-16T07:37:57.204 INFO:teuthology.orchestra.run.smithi070.stdout:(30/32): librbd1-19.0.0-3109.g30b0aa00.el9.x86_ 3.6 MB/s | 3.7 MB 00:01 2024-04-16T07:37:59.187 INFO:teuthology.orchestra.run.smithi070.stdout:(31/32): ceph-common-19.0.0-3109.g30b0aa00.el9. 1.2 MB/s | 19 MB 00:16 2024-04-16T07:37:59.636 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:37:59.650 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:37:59.650 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-16T07:37:59.650 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:37:59.650 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-16T07:37:59.650 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 10 M 2024-04-16T07:37:59.650 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrading: 2024-04-16T07:37:59.650 INFO:teuthology.orchestra.run.smithi172.stdout: librados2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.4 M 2024-04-16T07:37:59.650 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.7 M 2024-04-16T07:37:59.651 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-16T07:37:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: boost-program-options 2024-04-16T07:37:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T07:37:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.9 M 2024-04-16T07:37:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 19 M 2024-04-16T07:37:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 25 k 2024-04-16T07:37:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T07:37:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T07:37:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T07:37:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 797 k 2024-04-16T07:37:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T07:37:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T07:37:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T07:37:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 546 k 2024-04-16T07:37:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T07:37:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.2 M 2024-04-16T07:37:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T07:37:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T07:37:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T07:37:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse 2024-04-16T07:37:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 45 k 2024-04-16T07:37:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 135 k 2024-04-16T07:37:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 159 k 2024-04-16T07:37:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T07:37:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 315 k 2024-04-16T07:37:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 293 k 2024-04-16T07:37:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 99 k 2024-04-16T07:37:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-16T07:37:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-16T07:37:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T07:37:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T07:37:59.653 INFO:teuthology.orchestra.run.smithi172.stdout:Installing weak dependencies: 2024-04-16T07:37:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-16T07:37:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:37:59.653 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-16T07:37:59.653 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:37:59.653 INFO:teuthology.orchestra.run.smithi172.stdout:Install 30 Packages 2024-04-16T07:37:59.653 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrade 2 Packages 2024-04-16T07:37:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:37:59.654 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 59 M 2024-04-16T07:37:59.654 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-16T07:38:01.995 INFO:teuthology.orchestra.run.smithi070.stdout:(32/32): ceph-radosgw-19.0.0-3109.g30b0aa00.el9 547 kB/s | 10 MB 00:18 2024-04-16T07:38:01.998 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:38:01.998 INFO:teuthology.orchestra.run.smithi070.stdout:Total 3.1 MB/s | 59 MB 00:19 2024-04-16T07:38:01.999 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-16T07:38:02.036 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-16T07:38:02.036 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-16T07:38:02.521 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-16T07:38:02.521 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-16T07:38:04.020 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-16T07:38:04.516 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-16T07:38:04.605 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:38:04.678 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:38:04.768 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/34 2024-04-16T07:38:04.789 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:38:04.844 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:38:04.891 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-16T07:38:04.919 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-16T07:38:04.944 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-16T07:38:04.967 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T07:38:04.983 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T07:38:05.169 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T07:38:05.199 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T07:38:05.547 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-16T07:38:05.579 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:38:05.629 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:38:05.672 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9. 12/34 2024-04-16T07:38:05.727 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 13/34 2024-04-16T07:38:05.759 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 14/34 2024-04-16T07:38:05.839 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-16T07:38:05.873 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 16/34 2024-04-16T07:38:05.981 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 16/34 2024-04-16T07:38:06.014 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-16T07:38:06.017 INFO:teuthology.orchestra.run.smithi172.stdout:(1/32): ceph-base-19.0.0-3109.g30b0aa00.el9.x86 978 kB/s | 5.9 MB 00:06 2024-04-16T07:38:06.117 INFO:teuthology.orchestra.run.smithi172.stdout:(2/32): ceph-selinux-19.0.0-3109.g30b0aa00.el9. 252 kB/s | 25 kB 00:00 2024-04-16T07:38:06.133 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-16T07:38:06.164 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-16T07:38:06.205 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-16T07:38:06.246 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-16T07:38:06.274 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-16T07:38:06.306 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-16T07:38:06.349 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-16T07:38:06.917 INFO:teuthology.orchestra.run.smithi172.stdout:(3/32): libcephfs2-19.0.0-3109.g30b0aa00.el9.x8 996 kB/s | 797 kB 00:00 2024-04-16T07:38:06.929 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-16T07:38:06.962 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 26/34 2024-04-16T07:38:07.016 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 26/34 2024-04-16T07:38:07.059 INFO:teuthology.orchestra.run.smithi172.stdout:(4/32): ceph-radosgw-19.0.0-3109.g30b0aa00.el9. 1.4 MB/s | 10 MB 00:07 2024-04-16T07:38:07.085 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 27/34 2024-04-16T07:38:07.114 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x8 28/34 2024-04-16T07:38:07.568 INFO:teuthology.orchestra.run.smithi172.stdout:(5/32): libradosstriper1-19.0.0-3109.g30b0aa00. 841 kB/s | 546 kB 00:00 2024-04-16T07:38:07.709 INFO:teuthology.orchestra.run.smithi172.stdout:(6/32): python3-ceph-argparse-19.0.0-3109.g30b0 319 kB/s | 45 kB 00:00 2024-04-16T07:38:07.935 INFO:teuthology.orchestra.run.smithi172.stdout:(7/32): python3-ceph-common-19.0.0-3109.g30b0aa 599 kB/s | 135 kB 00:00 2024-04-16T07:38:08.135 INFO:teuthology.orchestra.run.smithi172.stdout:(8/32): python3-cephfs-19.0.0-3109.g30b0aa00.el 798 kB/s | 159 kB 00:00 2024-04-16T07:38:08.360 INFO:teuthology.orchestra.run.smithi172.stdout:(9/32): ceph-common-19.0.0-3109.g30b0aa00.el9.x 2.3 MB/s | 19 MB 00:08 2024-04-16T07:38:08.477 INFO:teuthology.orchestra.run.smithi172.stdout:(10/32): python3-rados-19.0.0-3109.g30b0aa00.el 922 kB/s | 315 kB 00:00 2024-04-16T07:38:08.602 INFO:teuthology.orchestra.run.smithi172.stdout:(11/32): python3-rgw-19.0.0-3109.g30b0aa00.el9. 796 kB/s | 99 kB 00:00 2024-04-16T07:38:08.628 INFO:teuthology.orchestra.run.smithi172.stdout:(12/32): python3-rbd-19.0.0-3109.g30b0aa00.el9. 1.1 MB/s | 293 kB 00:00 2024-04-16T07:38:08.761 INFO:teuthology.orchestra.run.smithi172.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 668 kB/s | 106 kB 00:00 2024-04-16T07:38:08.786 INFO:teuthology.orchestra.run.smithi172.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-04-16T07:38:08.812 INFO:teuthology.orchestra.run.smithi172.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-04-16T07:38:08.887 INFO:teuthology.orchestra.run.smithi172.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 661 kB 00:00 2024-04-16T07:38:08.912 INFO:teuthology.orchestra.run.smithi172.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-04-16T07:38:08.938 INFO:teuthology.orchestra.run.smithi172.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 825 kB/s | 41 kB 00:00 2024-04-16T07:38:08.963 INFO:teuthology.orchestra.run.smithi172.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-04-16T07:38:08.988 INFO:teuthology.orchestra.run.smithi172.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 698 kB/s | 35 kB 00:00 2024-04-16T07:38:09.013 INFO:teuthology.orchestra.run.smithi172.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 924 kB/s | 46 kB 00:00 2024-04-16T07:38:09.280 INFO:teuthology.orchestra.run.smithi172.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-04-16T07:38:09.339 INFO:teuthology.orchestra.run.smithi172.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 444 kB/s | 26 kB 00:00 2024-04-16T07:38:09.406 INFO:teuthology.orchestra.run.smithi172.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 730 kB/s | 49 kB 00:00 2024-04-16T07:38:09.506 INFO:teuthology.orchestra.run.smithi172.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 9.0 MB/s | 4.4 MB 00:00 2024-04-16T07:38:09.531 INFO:teuthology.orchestra.run.smithi172.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 537 kB/s | 67 kB 00:00 2024-04-16T07:38:09.590 INFO:teuthology.orchestra.run.smithi172.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-04-16T07:38:09.594 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:38:09.615 INFO:teuthology.orchestra.run.smithi172.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-16T07:38:09.643 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:38:09.682 INFO:teuthology.orchestra.run.smithi172.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-04-16T07:38:10.338 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:38:10.490 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 30/34 2024-04-16T07:38:10.580 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 30/34 2024-04-16T07:38:10.580 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T07:38:10.580 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T07:38:10.580 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:38:10.599 INFO:teuthology.orchestra.run.smithi172.stdout:(30/32): librgw2-19.0.0-3109.g30b0aa00.el9.x86_ 1.5 MB/s | 5.2 MB 00:03 2024-04-16T07:38:10.615 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:38:12.933 INFO:teuthology.orchestra.run.smithi172.stdout:(31/32): librados2-19.0.0-3109.g30b0aa00.el9.x8 1.0 MB/s | 3.4 MB 00:03 2024-04-16T07:38:13.799 INFO:teuthology.orchestra.run.smithi172.stdout:(32/32): librbd1-19.0.0-3109.g30b0aa00.el9.x86_ 914 kB/s | 3.7 MB 00:04 2024-04-16T07:38:13.805 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:38:13.805 INFO:teuthology.orchestra.run.smithi172.stdout:Total 4.2 MB/s | 59 MB 00:14 2024-04-16T07:38:13.806 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-16T07:38:13.843 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-16T07:38:13.843 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-16T07:38:14.324 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-16T07:38:14.324 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-16T07:38:15.807 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-16T07:38:16.307 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-16T07:38:16.418 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:38:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:38:16.668 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/34 2024-04-16T07:38:16.694 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:38:16.741 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:38:16.785 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-16T07:38:16.816 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-16T07:38:16.841 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-16T07:38:16.864 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T07:38:16.880 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T07:38:17.066 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T07:38:17.096 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T07:38:17.443 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-16T07:38:17.476 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:38:17.526 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:38:17.569 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9. 12/34 2024-04-16T07:38:17.623 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 13/34 2024-04-16T07:38:17.648 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 14/34 2024-04-16T07:38:17.727 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-16T07:38:17.754 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 16/34 2024-04-16T07:38:17.842 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 16/34 2024-04-16T07:38:17.878 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-16T07:38:17.986 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-16T07:38:18.021 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-16T07:38:18.061 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-16T07:38:18.085 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-16T07:38:18.114 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-16T07:38:18.145 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-16T07:38:18.191 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-16T07:38:18.751 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-16T07:38:18.777 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 26/34 2024-04-16T07:38:18.831 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 26/34 2024-04-16T07:38:18.899 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 27/34 2024-04-16T07:38:18.929 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x8 28/34 2024-04-16T07:38:20.287 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:38:20.287 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /sys 2024-04-16T07:38:20.287 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /proc 2024-04-16T07:38:20.287 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /mnt 2024-04-16T07:38:20.288 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /var/tmp 2024-04-16T07:38:20.288 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /home 2024-04-16T07:38:20.288 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /root 2024-04-16T07:38:20.288 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /tmp 2024-04-16T07:38:20.288 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:38:21.349 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:38:21.379 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/34 2024-04-16T07:38:21.392 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:38:21.405 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/34 2024-04-16T07:38:21.405 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:38:21.405 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T07:38:21.405 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:38:21.405 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:38:21.405 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:38:21.406 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T07:38:21.490 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T07:38:21.490 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:38:22.076 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:38:22.112 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 30/34 2024-04-16T07:38:22.219 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 30/34 2024-04-16T07:38:22.219 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T07:38:22.219 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T07:38:22.219 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:38:22.255 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:38:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:38:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/34 2024-04-16T07:38:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:38:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/34 2024-04-16T07:38:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:38:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 5/34 2024-04-16T07:38:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 6/34 2024-04-16T07:38:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 7/34 2024-04-16T07:38:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9. 8/34 2024-04-16T07:38:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x8 9/34 2024-04-16T07:38:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/34 2024-04-16T07:38:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:38:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 12/34 2024-04-16T07:38:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 13/34 2024-04-16T07:38:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-16T07:38:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-16T07:38:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-16T07:38:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-16T07:38:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-16T07:38:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-16T07:38:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-16T07:38:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-16T07:38:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-16T07:38:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-16T07:38:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-16T07:38:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-16T07:38:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-16T07:38:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-16T07:38:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-16T07:38:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-16T07:38:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-16T07:38:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:38:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-16T07:38:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/34 2024-04-16T07:38:23.996 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:38:23.996 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:38:23.996 INFO:teuthology.orchestra.run.smithi070.stdout:Upgraded: 2024-04-16T07:38:23.996 INFO:teuthology.orchestra.run.smithi070.stdout: librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:23.996 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:23.996 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-16T07:38:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T07:38:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T07:38:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T07:38:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T07:38:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T07:38:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T07:38:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T07:38:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T07:38:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T07:38:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T07:38:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T07:38:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T07:38:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-16T07:38:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-16T07:38:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-16T07:38:23.999 INFO:teuthology.orchestra.run.smithi070.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T07:38:23.999 INFO:teuthology.orchestra.run.smithi070.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T07:38:23.999 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:38:23.999 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:38:24.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-test 2024-04-16T07:38:24.908 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:45 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:38:25.063 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:38:25.065 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:38:25.065 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-04-16T07:38:25.065 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:38:25.065 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-16T07:38:25.065 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 84 M 2024-04-16T07:38:25.065 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-04-16T07:38:25.065 INFO:teuthology.orchestra.run.smithi070.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T07:38:25.065 INFO:teuthology.orchestra.run.smithi070.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-16T07:38:25.065 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 173 k 2024-04-16T07:38:25.066 INFO:teuthology.orchestra.run.smithi070.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-16T07:38:25.066 INFO:teuthology.orchestra.run.smithi070.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-04-16T07:38:25.066 INFO:teuthology.orchestra.run.smithi070.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T07:38:25.066 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-16T07:38:25.066 INFO:teuthology.orchestra.run.smithi070.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T07:38:25.066 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T07:38:25.066 INFO:teuthology.orchestra.run.smithi070.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-16T07:38:25.066 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:38:25.066 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-16T07:38:25.066 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:38:25.066 INFO:teuthology.orchestra.run.smithi070.stdout:Install 11 Packages 2024-04-16T07:38:25.066 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:38:25.067 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 87 M 2024-04-16T07:38:25.067 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 814 M 2024-04-16T07:38:25.067 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-16T07:38:25.692 INFO:teuthology.orchestra.run.smithi070.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-04-16T07:38:25.809 INFO:teuthology.orchestra.run.smithi070.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-04-16T07:38:25.867 INFO:teuthology.orchestra.run.smithi070.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-04-16T07:38:25.917 INFO:teuthology.orchestra.run.smithi070.stdout:(4/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-04-16T07:38:25.984 INFO:teuthology.orchestra.run.smithi070.stdout:(5/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 1.3 MB/s | 90 kB 00:00 2024-04-16T07:38:26.143 INFO:teuthology.orchestra.run.smithi070.stdout:(6/11): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-04-16T07:38:26.210 INFO:teuthology.orchestra.run.smithi070.stdout:(7/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 6.9 MB/s | 469 kB 00:00 2024-04-16T07:38:26.243 INFO:teuthology.orchestra.run.smithi070.stdout:(8/11): libubsan-11.4.1-3.el9.x86_64.rpm 5.5 MB/s | 187 kB 00:00 2024-04-16T07:38:26.410 INFO:teuthology.orchestra.run.smithi070.stdout:(9/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 730 kB/s | 122 kB 00:00 2024-04-16T07:38:26.452 INFO:teuthology.orchestra.run.smithi070.stdout:(10/11): libcephsqlite-19.0.0-3109.g30b0aa00.el 191 kB/s | 173 kB 00:00 2024-04-16T07:38:31.830 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:38:31.831 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /sys 2024-04-16T07:38:31.831 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /proc 2024-04-16T07:38:31.831 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /mnt 2024-04-16T07:38:31.831 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /var/tmp 2024-04-16T07:38:31.831 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /home 2024-04-16T07:38:31.831 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /root 2024-04-16T07:38:31.831 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /tmp 2024-04-16T07:38:31.831 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:38:32.874 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/34 2024-04-16T07:38:32.900 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/34 2024-04-16T07:38:32.900 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:38:32.900 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T07:38:32.901 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:38:32.901 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:38:32.901 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:38:32.902 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T07:38:32.985 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T07:38:32.985 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:38:34.218 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:38:34.218 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/34 2024-04-16T07:38:34.218 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:38:34.219 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/34 2024-04-16T07:38:34.219 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:38:34.219 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 5/34 2024-04-16T07:38:34.219 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 6/34 2024-04-16T07:38:34.219 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 7/34 2024-04-16T07:38:34.219 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9. 8/34 2024-04-16T07:38:34.219 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x8 9/34 2024-04-16T07:38:34.219 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/34 2024-04-16T07:38:34.219 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:38:34.219 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 12/34 2024-04-16T07:38:34.219 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 13/34 2024-04-16T07:38:34.220 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-16T07:38:34.220 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-16T07:38:34.220 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-16T07:38:34.220 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-16T07:38:34.220 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-16T07:38:34.220 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-16T07:38:34.220 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-16T07:38:34.220 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-16T07:38:34.220 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-16T07:38:34.220 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-16T07:38:34.220 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-16T07:38:34.220 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-16T07:38:34.220 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-16T07:38:34.220 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-16T07:38:34.221 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-16T07:38:34.221 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-16T07:38:34.221 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-16T07:38:34.221 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:38:34.221 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-16T07:38:34.221 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/34 2024-04-16T07:38:35.282 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:38:35.282 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:38:35.282 INFO:teuthology.orchestra.run.smithi172.stdout:Upgraded: 2024-04-16T07:38:35.282 INFO:teuthology.orchestra.run.smithi172.stdout: librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:35.282 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:35.282 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-16T07:38:35.282 INFO:teuthology.orchestra.run.smithi172.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T07:38:35.282 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:35.282 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:35.283 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:35.283 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:35.283 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T07:38:35.283 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T07:38:35.283 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T07:38:35.283 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:35.283 INFO:teuthology.orchestra.run.smithi172.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T07:38:35.283 INFO:teuthology.orchestra.run.smithi172.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T07:38:35.283 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T07:38:35.283 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:35.283 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T07:38:35.284 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:35.284 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T07:38:35.284 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T07:38:35.284 INFO:teuthology.orchestra.run.smithi172.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T07:38:35.284 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:35.284 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:35.284 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:35.284 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T07:38:35.284 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:35.284 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:35.284 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:35.284 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-16T07:38:35.285 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-16T07:38:35.285 INFO:teuthology.orchestra.run.smithi172.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-16T07:38:35.285 INFO:teuthology.orchestra.run.smithi172.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T07:38:35.285 INFO:teuthology.orchestra.run.smithi172.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T07:38:35.285 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:38:35.285 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:38:35.644 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-test 2024-04-16T07:38:36.223 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:40 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:38:36.382 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:38:36.383 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:38:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-16T07:38:36.383 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:38:36.383 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-16T07:38:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 84 M 2024-04-16T07:38:36.383 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-16T07:38:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T07:38:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-16T07:38:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 173 k 2024-04-16T07:38:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-16T07:38:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-04-16T07:38:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T07:38:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-16T07:38:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T07:38:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T07:38:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-16T07:38:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:38:36.384 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-16T07:38:36.384 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:38:36.384 INFO:teuthology.orchestra.run.smithi172.stdout:Install 11 Packages 2024-04-16T07:38:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:38:36.385 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 87 M 2024-04-16T07:38:36.385 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 814 M 2024-04-16T07:38:36.385 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-16T07:38:36.935 INFO:teuthology.orchestra.run.smithi172.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 917 kB/s | 217 kB 00:00 2024-04-16T07:38:37.361 INFO:teuthology.orchestra.run.smithi172.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 2.4 MB/s | 1.0 MB 00:00 2024-04-16T07:38:37.394 INFO:teuthology.orchestra.run.smithi172.stdout:(3/11): libcephsqlite-19.0.0-3109.g30b0aa00.el9 249 kB/s | 173 kB 00:00 2024-04-16T07:38:37.503 INFO:teuthology.orchestra.run.smithi172.stdout:(4/11): socat-1.7.4.1-5.el9.x86_64.rpm 2.1 MB/s | 300 kB 00:00 2024-04-16T07:38:37.528 INFO:teuthology.orchestra.run.smithi172.stdout:(5/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 468 kB/s | 63 kB 00:00 2024-04-16T07:38:37.570 INFO:teuthology.orchestra.run.smithi172.stdout:(6/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 1.3 MB/s | 90 kB 00:00 2024-04-16T07:38:37.854 INFO:teuthology.orchestra.run.smithi172.stdout:(7/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 1.6 MB/s | 469 kB 00:00 2024-04-16T07:38:37.879 INFO:teuthology.orchestra.run.smithi172.stdout:(8/11): jq-1.6-16.el9.x86_64.rpm 534 kB/s | 187 kB 00:00 2024-04-16T07:38:37.954 INFO:teuthology.orchestra.run.smithi172.stdout:(9/11): libubsan-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2024-04-16T07:38:38.063 INFO:teuthology.orchestra.run.smithi172.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 662 kB/s | 122 kB 00:00 2024-04-16T07:39:58.735 INFO:teuthology.orchestra.run.smithi172.stdout:(11/11): ceph-test-19.0.0-3109.g30b0aa00.el9.x8 1.0 MB/s | 84 MB 01:22 2024-04-16T07:39:58.738 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:39:58.738 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.1 MB/s | 87 MB 01:22 2024-04-16T07:39:58.774 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-16T07:39:58.798 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-16T07:39:58.798 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-16T07:39:59.422 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-16T07:39:59.422 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-16T07:40:00.426 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-16T07:40:00.490 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-04-16T07:40:00.527 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-04-16T07:40:00.552 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T07:40:00.615 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T07:40:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-04-16T07:40:00.687 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-04-16T07:40:00.750 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-04-16T07:40:00.781 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-04-16T07:40:00.815 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-04-16T07:40:00.857 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-04-16T07:40:00.883 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/11 2024-04-16T07:40:12.317 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/11 2024-04-16T07:40:12.389 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/11 2024-04-16T07:40:18.417 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/11 2024-04-16T07:40:18.417 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/11 2024-04-16T07:40:18.417 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/11 2024-04-16T07:40:18.417 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-04-16T07:40:18.417 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-04-16T07:40:18.418 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-04-16T07:40:18.418 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-04-16T07:40:18.418 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-04-16T07:40:18.418 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-04-16T07:40:18.418 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-04-16T07:40:18.418 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-04-16T07:40:18.850 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-04-16T07:40:18.850 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:40:18.850 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-16T07:40:18.850 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:40:18.850 INFO:teuthology.orchestra.run.smithi172.stdout: jq-1.6-16.el9.x86_64 2024-04-16T07:40:18.850 INFO:teuthology.orchestra.run.smithi172.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-16T07:40:18.850 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:40:18.850 INFO:teuthology.orchestra.run.smithi172.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-16T07:40:18.850 INFO:teuthology.orchestra.run.smithi172.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-16T07:40:18.850 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T07:40:18.850 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-16T07:40:18.850 INFO:teuthology.orchestra.run.smithi172.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T07:40:18.851 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T07:40:18.851 INFO:teuthology.orchestra.run.smithi172.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-16T07:40:18.851 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:40:18.851 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:40:19.148 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph 2024-04-16T07:40:19.733 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:23 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:40:19.888 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:40:19.890 INFO:teuthology.orchestra.run.smithi172.stdout:========================================================================================= 2024-04-16T07:40:19.890 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-16T07:40:19.891 INFO:teuthology.orchestra.run.smithi172.stdout:========================================================================================= 2024-04-16T07:40:19.891 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-16T07:40:19.891 INFO:teuthology.orchestra.run.smithi172.stdout: ceph x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 6.5 k 2024-04-16T07:40:19.891 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-16T07:40:19.891 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 2.2 M 2024-04-16T07:40:19.891 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 1.6 M 2024-04-16T07:40:19.891 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 249 k 2024-04-16T07:40:19.891 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.0 M 2024-04-16T07:40:19.891 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 71 M 2024-04-16T07:40:19.891 INFO:teuthology.orchestra.run.smithi172.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-16T07:40:19.891 INFO:teuthology.orchestra.run.smithi172.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-16T07:40:19.891 INFO:teuthology.orchestra.run.smithi172.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-16T07:40:19.891 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T07:40:19.891 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T07:40:19.892 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T07:40:19.892 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T07:40:19.892 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T07:40:19.892 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T07:40:19.892 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T07:40:19.892 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T07:40:19.892 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T07:40:19.892 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T07:40:19.892 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T07:40:19.892 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T07:40:19.892 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T07:40:19.892 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T07:40:19.893 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T07:40:19.893 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T07:40:19.893 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T07:40:19.893 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T07:40:19.893 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T07:40:19.893 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T07:40:19.893 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T07:40:19.893 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T07:40:19.893 INFO:teuthology.orchestra.run.smithi172.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T07:40:19.893 INFO:teuthology.orchestra.run.smithi172.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T07:40:19.893 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T07:40:19.893 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T07:40:19.893 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T07:40:19.893 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:40:19.893 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-16T07:40:19.894 INFO:teuthology.orchestra.run.smithi172.stdout:========================================================================================= 2024-04-16T07:40:19.894 INFO:teuthology.orchestra.run.smithi172.stdout:Install 36 Packages 2024-04-16T07:40:19.894 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:40:19.895 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 84 M 2024-04-16T07:40:19.895 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 922 M 2024-04-16T07:40:19.895 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-16T07:40:20.730 INFO:teuthology.orchestra.run.smithi172.stdout:(1/36): ceph-19.0.0-3109.g30b0aa00.el9.x86_64.r 19 kB/s | 6.5 kB 00:00 2024-04-16T07:40:22.239 INFO:teuthology.orchestra.run.smithi172.stdout:(2/36): ceph-mds-19.0.0-3109.g30b0aa00.el9.x86_ 1.2 MB/s | 2.2 MB 00:01 2024-04-16T07:40:23.272 INFO:teuthology.orchestra.run.smithi172.stdout:(3/36): ceph-mgr-19.0.0-3109.g30b0aa00.el9.x86_ 552 kB/s | 1.6 MB 00:02 2024-04-16T07:40:23.747 INFO:teuthology.orchestra.run.smithi172.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3109.g30b0 524 kB/s | 249 kB 00:00 2024-04-16T07:40:24.056 INFO:teuthology.orchestra.run.smithi172.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 606 kB/s | 187 kB 00:00 2024-04-16T07:40:24.156 INFO:teuthology.orchestra.run.smithi172.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2024-04-16T07:40:24.215 INFO:teuthology.orchestra.run.smithi172.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 541 kB/s | 32 kB 00:00 2024-04-16T07:40:24.282 INFO:teuthology.orchestra.run.smithi172.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 711 kB/s | 47 kB 00:00 2024-04-16T07:40:24.340 INFO:teuthology.orchestra.run.smithi172.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 663 kB/s | 39 kB 00:00 2024-04-16T07:40:24.474 INFO:teuthology.orchestra.run.smithi172.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.8 MB/s | 241 kB 00:00 2024-04-16T07:40:24.608 INFO:teuthology.orchestra.run.smithi172.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 8.6 MB/s | 1.1 MB 00:00 2024-04-16T07:40:24.649 INFO:teuthology.orchestra.run.smithi172.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-04-16T07:40:24.691 INFO:teuthology.orchestra.run.smithi172.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-04-16T07:40:24.733 INFO:teuthology.orchestra.run.smithi172.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-04-16T07:40:25.250 INFO:teuthology.orchestra.run.smithi172.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 242 kB/s | 125 kB 00:00 2024-04-16T07:40:25.325 INFO:teuthology.orchestra.run.smithi172.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 298 kB/s | 22 kB 00:00 2024-04-16T07:40:25.734 INFO:teuthology.orchestra.run.smithi172.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 370 kB/s | 151 kB 00:00 2024-04-16T07:40:25.834 INFO:teuthology.orchestra.run.smithi172.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 435 kB/s | 43 kB 00:00 2024-04-16T07:40:25.985 INFO:teuthology.orchestra.run.smithi172.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-04-16T07:40:26.160 INFO:teuthology.orchestra.run.smithi172.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.0 MB/s | 358 kB 00:00 2024-04-16T07:40:26.235 INFO:teuthology.orchestra.run.smithi172.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 143 kB/s | 11 kB 00:00 2024-04-16T07:40:26.310 INFO:teuthology.orchestra.run.smithi172.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-04-16T07:40:26.386 INFO:teuthology.orchestra.run.smithi172.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 309 kB/s | 23 kB 00:00 2024-04-16T07:40:26.461 INFO:teuthology.orchestra.run.smithi172.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-04-16T07:40:26.553 INFO:teuthology.orchestra.run.smithi172.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 215 kB/s | 20 kB 00:00 2024-04-16T07:40:26.637 INFO:teuthology.orchestra.run.smithi172.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 556 kB/s | 46 kB 00:00 2024-04-16T07:40:26.712 INFO:teuthology.orchestra.run.smithi172.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-04-16T07:40:26.821 INFO:teuthology.orchestra.run.smithi172.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-04-16T07:40:26.904 INFO:teuthology.orchestra.run.smithi172.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 197 kB/s | 16 kB 00:00 2024-04-16T07:40:26.988 INFO:teuthology.orchestra.run.smithi172.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-04-16T07:40:27.063 INFO:teuthology.orchestra.run.smithi172.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 477 kB/s | 36 kB 00:00 2024-04-16T07:40:27.155 INFO:teuthology.orchestra.run.smithi172.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-16T07:40:27.272 INFO:teuthology.orchestra.run.smithi172.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-04-16T07:40:27.347 INFO:teuthology.orchestra.run.smithi172.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-04-16T07:40:30.514 INFO:teuthology.orchestra.run.smithi172.stdout:(35/36): ceph-mon-19.0.0-3109.g30b0aa00.el9.x86 526 kB/s | 5.0 MB 00:09 2024-04-16T07:40:48.100 INFO:teuthology.orchestra.run.smithi070.stdout:(11/11): ceph-test-19.0.0-3109.g30b0aa00.el9.x8 606 kB/s | 84 MB 02:22 2024-04-16T07:40:48.102 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:40:48.102 INFO:teuthology.orchestra.run.smithi070.stdout:Total 623 kB/s | 87 MB 02:23 2024-04-16T07:40:48.136 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-16T07:40:48.158 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-16T07:40:48.158 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-16T07:40:48.777 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-16T07:40:48.778 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-16T07:40:49.812 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-16T07:40:49.886 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-04-16T07:40:49.917 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-04-16T07:40:49.951 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T07:40:50.008 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T07:40:50.043 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-04-16T07:40:50.095 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-04-16T07:40:50.139 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-04-16T07:40:50.171 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-04-16T07:40:50.211 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-04-16T07:40:50.254 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-04-16T07:40:50.281 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/11 2024-04-16T07:41:01.548 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/11 2024-04-16T07:41:01.620 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/11 2024-04-16T07:41:07.804 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/11 2024-04-16T07:41:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/11 2024-04-16T07:41:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/11 2024-04-16T07:41:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-04-16T07:41:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-04-16T07:41:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-04-16T07:41:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-04-16T07:41:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-04-16T07:41:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-04-16T07:41:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-04-16T07:41:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-04-16T07:41:08.223 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-04-16T07:41:08.223 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:41:08.223 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-16T07:41:08.223 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:41:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: jq-1.6-16.el9.x86_64 2024-04-16T07:41:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-16T07:41:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:41:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-16T07:41:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-16T07:41:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T07:41:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-16T07:41:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T07:41:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T07:41:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-16T07:41:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:41:08.225 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:41:08.478 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph 2024-04-16T07:41:09.057 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:03:30 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:41:09.208 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:41:09.210 INFO:teuthology.orchestra.run.smithi070.stdout:========================================================================================= 2024-04-16T07:41:09.211 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-04-16T07:41:09.211 INFO:teuthology.orchestra.run.smithi070.stdout:========================================================================================= 2024-04-16T07:41:09.211 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-16T07:41:09.211 INFO:teuthology.orchestra.run.smithi070.stdout: ceph x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 6.5 k 2024-04-16T07:41:09.211 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-04-16T07:41:09.211 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 2.2 M 2024-04-16T07:41:09.211 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 1.6 M 2024-04-16T07:41:09.211 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 249 k 2024-04-16T07:41:09.211 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.0 M 2024-04-16T07:41:09.211 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 71 M 2024-04-16T07:41:09.211 INFO:teuthology.orchestra.run.smithi070.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-16T07:41:09.211 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-16T07:41:09.211 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-16T07:41:09.211 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T07:41:09.212 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T07:41:09.212 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T07:41:09.212 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T07:41:09.212 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T07:41:09.212 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T07:41:09.212 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T07:41:09.212 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T07:41:09.212 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T07:41:09.212 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T07:41:09.212 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T07:41:09.212 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T07:41:09.212 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T07:41:09.213 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T07:41:09.213 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T07:41:09.213 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T07:41:09.213 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T07:41:09.213 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T07:41:09.213 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T07:41:09.213 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T07:41:09.213 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T07:41:09.213 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T07:41:09.213 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T07:41:09.213 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T07:41:09.213 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T07:41:09.213 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T07:41:09.213 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T07:41:09.213 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:41:09.213 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-16T07:41:09.214 INFO:teuthology.orchestra.run.smithi070.stdout:========================================================================================= 2024-04-16T07:41:09.214 INFO:teuthology.orchestra.run.smithi070.stdout:Install 36 Packages 2024-04-16T07:41:09.214 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:41:09.214 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 84 M 2024-04-16T07:41:09.214 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 922 M 2024-04-16T07:41:09.214 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-16T07:41:10.178 INFO:teuthology.orchestra.run.smithi070.stdout:(1/36): ceph-19.0.0-3109.g30b0aa00.el9.x86_64.r 15 kB/s | 6.5 kB 00:00 2024-04-16T07:41:12.337 INFO:teuthology.orchestra.run.smithi070.stdout:(2/36): ceph-mgr-19.0.0-3109.g30b0aa00.el9.x86_ 618 kB/s | 1.6 MB 00:02 2024-04-16T07:41:13.320 INFO:teuthology.orchestra.run.smithi070.stdout:(3/36): ceph-mds-19.0.0-3109.g30b0aa00.el9.x86_ 627 kB/s | 2.2 MB 00:03 2024-04-16T07:41:13.522 INFO:teuthology.orchestra.run.smithi070.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3109.g30b0 1.2 MB/s | 249 kB 00:00 2024-04-16T07:41:13.631 INFO:teuthology.orchestra.run.smithi070.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-04-16T07:41:13.664 INFO:teuthology.orchestra.run.smithi070.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-04-16T07:41:13.698 INFO:teuthology.orchestra.run.smithi070.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 949 kB/s | 32 kB 00:00 2024-04-16T07:41:13.732 INFO:teuthology.orchestra.run.smithi070.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-04-16T07:41:13.765 INFO:teuthology.orchestra.run.smithi070.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-04-16T07:41:13.807 INFO:teuthology.orchestra.run.smithi070.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-04-16T07:41:13.941 INFO:teuthology.orchestra.run.smithi070.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 8.6 MB/s | 1.1 MB 00:00 2024-04-16T07:41:13.999 INFO:teuthology.orchestra.run.smithi070.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 103 kB 00:00 2024-04-16T07:41:14.058 INFO:teuthology.orchestra.run.smithi070.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2024-04-16T07:41:14.108 INFO:teuthology.orchestra.run.smithi070.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-04-16T07:41:15.000 INFO:teuthology.orchestra.run.smithi070.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 140 kB/s | 125 kB 00:00 2024-04-16T07:41:15.084 INFO:teuthology.orchestra.run.smithi070.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-04-16T07:41:15.251 INFO:teuthology.orchestra.run.smithi070.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 907 kB/s | 151 kB 00:00 2024-04-16T07:41:15.301 INFO:teuthology.orchestra.run.smithi070.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 869 kB/s | 43 kB 00:00 2024-04-16T07:41:15.368 INFO:teuthology.orchestra.run.smithi070.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-04-16T07:41:15.451 INFO:teuthology.orchestra.run.smithi070.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2024-04-16T07:41:15.502 INFO:teuthology.orchestra.run.smithi070.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-04-16T07:41:15.552 INFO:teuthology.orchestra.run.smithi070.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 354 kB/s | 18 kB 00:00 2024-04-16T07:41:15.603 INFO:teuthology.orchestra.run.smithi070.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 465 kB/s | 23 kB 00:00 2024-04-16T07:41:15.653 INFO:teuthology.orchestra.run.smithi070.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-04-16T07:41:15.695 INFO:teuthology.orchestra.run.smithi070.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 476 kB/s | 20 kB 00:00 2024-04-16T07:41:15.737 INFO:teuthology.orchestra.run.smithi070.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-04-16T07:41:15.787 INFO:teuthology.orchestra.run.smithi070.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-04-16T07:41:15.846 INFO:teuthology.orchestra.run.smithi070.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2024-04-16T07:41:15.888 INFO:teuthology.orchestra.run.smithi070.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-04-16T07:41:15.929 INFO:teuthology.orchestra.run.smithi070.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-04-16T07:41:15.972 INFO:teuthology.orchestra.run.smithi070.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 858 kB/s | 36 kB 00:00 2024-04-16T07:41:16.039 INFO:teuthology.orchestra.run.smithi070.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2024-04-16T07:41:16.106 INFO:teuthology.orchestra.run.smithi070.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.3 MB/s | 427 kB 00:00 2024-04-16T07:41:16.148 INFO:teuthology.orchestra.run.smithi070.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 478 kB/s | 20 kB 00:00 2024-04-16T07:41:23.871 INFO:teuthology.orchestra.run.smithi070.stdout:(35/36): ceph-mon-19.0.0-3109.g30b0aa00.el9.x86 376 kB/s | 5.0 MB 00:13 2024-04-16T07:41:43.062 INFO:teuthology.orchestra.run.smithi172.stdout:(36/36): ceph-osd-19.0.0-3109.g30b0aa00.el9.x86 898 kB/s | 71 MB 01:20 2024-04-16T07:41:43.065 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:43.065 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.0 MB/s | 84 MB 01:23 2024-04-16T07:41:43.086 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-16T07:41:43.127 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-16T07:41:43.127 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-16T07:41:43.854 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-16T07:41:43.854 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-16T07:41:45.449 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-16T07:41:45.505 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-16T07:41:45.540 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-16T07:41:45.566 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-16T07:41:45.636 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-16T07:41:45.681 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-16T07:41:45.714 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-16T07:41:45.804 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-16T07:41:45.900 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-16T07:41:45.941 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-16T07:41:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-16T07:41:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-16T07:41:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-16T07:41:46.163 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-16T07:41:46.217 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-16T07:41:46.411 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-16T07:41:46.459 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-16T07:41:46.522 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-16T07:41:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-16T07:41:46.588 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-16T07:41:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-16T07:41:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T07:41:46.731 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-16T07:41:46.771 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-16T07:41:46.807 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-16T07:41:46.841 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-16T07:41:46.931 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-16T07:41:46.981 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-16T07:41:47.052 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-16T07:41:47.190 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-16T07:41:47.307 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T07:41:47.501 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9. 31/36 2024-04-16T07:41:47.535 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/36 2024-04-16T07:41:47.561 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/36 2024-04-16T07:41:47.561 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:41:47.561 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T07:41:47.561 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:41:47.561 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:41:47.562 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:41:58.428 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/36 2024-04-16T07:41:58.456 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/36 2024-04-16T07:41:58.456 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:41:58.457 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T07:41:58.457 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:41:58.457 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:41:58.457 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:41:58.995 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 34/36 2024-04-16T07:41:59.019 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 34/36 2024-04-16T07:41:59.019 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:41:59.019 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T07:41:59.019 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:41:59.019 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:41:59.019 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:41:59.254 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 35/36 2024-04-16T07:41:59.278 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 35/36 2024-04-16T07:41:59.278 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:41:59.278 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T07:41:59.278 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:41:59.278 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:41:59.278 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:41:59.353 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 36/36 2024-04-16T07:42:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 36/36 2024-04-16T07:42:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/36 2024-04-16T07:42:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/36 2024-04-16T07:42:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/36 2024-04-16T07:42:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/36 2024-04-16T07:42:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 5/36 2024-04-16T07:42:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9. 6/36 2024-04-16T07:42:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-16T07:42:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-16T07:42:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-16T07:42:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-16T07:42:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-16T07:42:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-16T07:42:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-16T07:42:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-16T07:42:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-16T07:42:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-16T07:42:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-16T07:42:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-16T07:42:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-16T07:42:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-16T07:42:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T07:42:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-16T07:42:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-16T07:42:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-16T07:42:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-16T07:42:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-16T07:42:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-16T07:42:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-16T07:42:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-16T07:42:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T07:42:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-16T07:42:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-16T07:42:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-16T07:42:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-16T07:42:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-16T07:42:06.763 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-16T07:42:06.764 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:42:06.764 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-16T07:42:06.764 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:42:06.764 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:42:06.764 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:42:06.764 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:42:06.764 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:42:06.764 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:42:06.764 INFO:teuthology.orchestra.run.smithi172.stdout: lua-5.4.4-4.el9.x86_64 2024-04-16T07:42:06.764 INFO:teuthology.orchestra.run.smithi172.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-16T07:42:06.764 INFO:teuthology.orchestra.run.smithi172.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-16T07:42:06.764 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T07:42:06.764 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T07:42:06.765 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T07:42:06.765 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T07:42:06.765 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T07:42:06.765 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T07:42:06.765 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T07:42:06.765 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T07:42:06.765 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T07:42:06.765 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T07:42:06.765 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T07:42:06.765 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T07:42:06.765 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T07:42:06.765 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T07:42:06.765 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T07:42:06.766 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T07:42:06.766 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T07:42:06.766 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T07:42:06.766 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T07:42:06.766 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T07:42:06.766 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T07:42:06.766 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T07:42:06.766 INFO:teuthology.orchestra.run.smithi172.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T07:42:06.766 INFO:teuthology.orchestra.run.smithi172.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T07:42:06.766 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T07:42:06.766 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T07:42:06.767 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T07:42:06.767 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:42:06.767 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:42:07.016 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-base 2024-04-16T07:42:07.597 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:11 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:42:07.696 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:42:07.747 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:42:07.748 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-16T07:42:07.748 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:42:07.816 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install cephadm 2024-04-16T07:42:08.400 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:12 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:42:08.550 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:42:08.551 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:42:08.551 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-16T07:42:08.551 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:42:08.551 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-16T07:42:08.551 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 762 k 2024-04-16T07:42:08.551 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:42:08.552 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-16T07:42:08.552 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:42:08.552 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-04-16T07:42:08.552 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:42:08.552 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 762 k 2024-04-16T07:42:08.552 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 769 k 2024-04-16T07:42:08.552 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-16T07:42:12.767 INFO:teuthology.orchestra.run.smithi172.stdout:cephadm-19.0.0-3109.g30b0aa00.el9.noarch.rpm 181 kB/s | 762 kB 00:04 2024-04-16T07:42:12.768 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:42:12.768 INFO:teuthology.orchestra.run.smithi172.stdout:Total 181 kB/s | 762 kB 00:04 2024-04-16T07:42:12.769 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-16T07:42:12.775 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-16T07:42:12.775 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-16T07:42:12.781 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-16T07:42:12.781 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-16T07:42:12.858 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-16T07:42:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:42:13.372 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:42:13.735 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:42:13.991 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:42:13.991 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:42:13.992 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-16T07:42:13.992 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:42:13.992 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:42:13.992 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:42:14.171 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T07:42:14.760 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:18 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:42:14.919 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:42:14.920 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:42:14.920 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-04-16T07:42:14.920 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:42:14.920 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-16T07:42:14.920 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 157 k 2024-04-16T07:42:14.920 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:42:14.920 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-16T07:42:14.920 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:42:14.920 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-04-16T07:42:14.920 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:42:14.920 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 157 k 2024-04-16T07:42:14.920 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 640 k 2024-04-16T07:42:14.920 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-16T07:42:16.239 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-immutable-object-cache-19.0.0-3109.g30b0aa 119 kB/s | 157 kB 00:01 2024-04-16T07:42:16.239 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:42:16.240 INFO:teuthology.orchestra.run.smithi172.stdout:Total 119 kB/s | 157 kB 00:01 2024-04-16T07:42:16.240 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-16T07:42:16.251 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-16T07:42:16.251 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-16T07:42:16.307 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-16T07:42:16.307 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-16T07:42:16.451 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-16T07:42:16.523 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:42:16.549 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:42:16.549 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:42:16.549 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T07:42:16.549 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:42:17.549 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:42:17.549 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:42:17.550 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-16T07:42:17.550 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:42:17.550 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:42:17.550 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:42:17.779 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr 2024-04-16T07:42:18.393 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:22 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:42:18.500 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:42:18.556 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:42:18.557 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-16T07:42:18.557 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:42:18.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T07:42:19.246 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:23 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:42:19.411 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:42:19.412 INFO:teuthology.orchestra.run.smithi172.stdout:====================================================================================== 2024-04-16T07:42:19.412 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-16T07:42:19.412 INFO:teuthology.orchestra.run.smithi172.stdout:====================================================================================== 2024-04-16T07:42:19.412 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-16T07:42:19.412 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 3.5 M 2024-04-16T07:42:19.412 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-16T07:42:19.412 INFO:teuthology.orchestra.run.smithi172.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-16T07:42:19.412 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 34 k 2024-04-16T07:42:19.412 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 16 k 2024-04-16T07:42:19.413 INFO:teuthology.orchestra.run.smithi172.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-16T07:42:19.413 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-16T07:42:19.413 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-16T07:42:19.413 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-16T07:42:19.413 INFO:teuthology.orchestra.run.smithi172.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-16T07:42:19.413 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T07:42:19.413 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T07:42:19.413 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:42:19.413 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-16T07:42:19.413 INFO:teuthology.orchestra.run.smithi172.stdout:====================================================================================== 2024-04-16T07:42:19.413 INFO:teuthology.orchestra.run.smithi172.stdout:Install 11 Packages 2024-04-16T07:42:19.413 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:42:19.414 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 7.6 M 2024-04-16T07:42:19.414 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 100 M 2024-04-16T07:42:19.414 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-16T07:42:20.276 INFO:teuthology.orchestra.run.smithi172.stdout:(1/11): ceph-prometheus-alerts-19.0.0-3109.g30b 27 kB/s | 16 kB 00:00 2024-04-16T07:42:20.476 INFO:teuthology.orchestra.run.smithi172.stdout:(2/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 238 kB 00:00 2024-04-16T07:42:20.626 INFO:teuthology.orchestra.run.smithi172.stdout:(3/11): ceph-grafana-dashboards-19.0.0-3109.g30 36 kB/s | 34 kB 00:00 2024-04-16T07:42:20.660 INFO:teuthology.orchestra.run.smithi172.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 4.6 MB/s | 863 kB 00:00 2024-04-16T07:42:21.043 INFO:teuthology.orchestra.run.smithi172.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 51 kB/s | 19 kB 00:00 2024-04-16T07:42:21.335 INFO:teuthology.orchestra.run.smithi172.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 773 kB/s | 548 kB 00:00 2024-04-16T07:42:21.444 INFO:teuthology.orchestra.run.smithi172.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-04-16T07:42:21.544 INFO:teuthology.orchestra.run.smithi172.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 307 kB/s | 31 kB 00:00 2024-04-16T07:42:21.678 INFO:teuthology.orchestra.run.smithi172.stdout:(9/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 3.2 MB/s | 2.0 MB 00:00 2024-04-16T07:42:21.702 INFO:teuthology.orchestra.run.smithi172.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 1.2 MB/s | 188 kB 00:00 2024-04-16T07:42:26.920 INFO:teuthology.orchestra.run.smithi172.stdout:(11/11): ceph-mgr-dashboard-19.0.0-3109.g30b0aa 500 kB/s | 3.5 MB 00:07 2024-04-16T07:42:26.925 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:42:26.925 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.0 MB/s | 7.6 MB 00:07 2024-04-16T07:42:26.945 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-16T07:42:26.965 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-16T07:42:26.965 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-16T07:42:27.135 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-16T07:42:27.136 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-16T07:42:27.613 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-16T07:42:27.709 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-16T07:42:27.739 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-16T07:42:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-16T07:42:27.955 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-16T07:42:28.000 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-16T07:42:28.084 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-16T07:42:28.135 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-16T07:42:28.164 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-16T07:42:28.197 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9 9/11 2024-04-16T07:42:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el 10/11 2024-04-16T07:42:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 11/11 2024-04-16T07:42:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 11/11 2024-04-16T07:42:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el 1/11 2024-04-16T07:42:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 2/11 2024-04-16T07:42:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9 3/11 2024-04-16T07:42:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-16T07:42:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-16T07:42:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-16T07:42:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-16T07:42:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-16T07:42:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-16T07:42:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-16T07:42:30.907 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-16T07:42:30.907 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:42:30.907 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-16T07:42:30.907 INFO:teuthology.orchestra.run.smithi172.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-16T07:42:30.907 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:42:30.907 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:42:30.907 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:42:30.907 INFO:teuthology.orchestra.run.smithi172.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-16T07:42:30.907 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-16T07:42:30.907 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-16T07:42:30.908 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-16T07:42:30.908 INFO:teuthology.orchestra.run.smithi172.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-16T07:42:30.908 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T07:42:30.908 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T07:42:30.908 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:42:30.908 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:42:31.185 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T07:42:31.766 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:35 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:42:31.921 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:42:31.922 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2024-04-16T07:42:31.922 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-16T07:42:31.923 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2024-04-16T07:42:31.923 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-16T07:42:31.923 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 7.4 M 2024-04-16T07:42:31.923 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-16T07:42:31.923 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T07:42:31.924 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T07:42:31.924 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T07:42:31.924 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T07:42:31.924 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T07:42:31.924 INFO:teuthology.orchestra.run.smithi172.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T07:42:31.924 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T07:42:31.925 INFO:teuthology.orchestra.run.smithi172.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T07:42:31.925 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T07:42:31.925 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T07:42:31.925 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T07:42:31.925 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:42:31.925 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-16T07:42:31.925 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2024-04-16T07:42:31.925 INFO:teuthology.orchestra.run.smithi172.stdout:Install 12 Packages 2024-04-16T07:42:31.925 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:42:31.926 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 37 M 2024-04-16T07:42:31.926 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 209 M 2024-04-16T07:42:31.926 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-16T07:42:32.634 INFO:teuthology.orchestra.run.smithi172.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 250 kB/s | 32 kB 00:00 2024-04-16T07:42:32.692 INFO:teuthology.orchestra.run.smithi172.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 504 kB/s | 17 kB 00:00 2024-04-16T07:42:32.726 INFO:teuthology.orchestra.run.smithi172.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-04-16T07:42:33.051 INFO:teuthology.orchestra.run.smithi172.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-04-16T07:42:33.218 INFO:teuthology.orchestra.run.smithi172.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-04-16T07:42:33.278 INFO:teuthology.orchestra.run.smithi172.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.9 MB/s | 3.0 MB 00:00 2024-04-16T07:42:33.302 INFO:teuthology.orchestra.run.smithi172.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-04-16T07:42:33.836 INFO:teuthology.orchestra.run.smithi172.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-04-16T07:42:33.953 INFO:teuthology.orchestra.run.smithi172.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-04-16T07:42:34.061 INFO:teuthology.orchestra.run.smithi172.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-04-16T07:42:34.086 INFO:teuthology.orchestra.run.smithi172.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 781 kB/s | 195 kB 00:00 2024-04-16T07:42:55.408 INFO:teuthology.orchestra.run.smithi172.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-3 331 kB/s | 7.4 MB 00:22 2024-04-16T07:42:55.411 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:42:55.411 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.6 MB/s | 37 MB 00:23 2024-04-16T07:42:55.455 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-16T07:42:55.482 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-16T07:42:55.482 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-16T07:42:55.838 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-16T07:42:55.839 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-16T07:42:56.550 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-16T07:42:56.614 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T07:42:56.646 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T07:42:56.750 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:42:56.782 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T07:42:56.959 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:42:56.981 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:42:57.057 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T07:42:57.713 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T07:42:57.836 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T07:42:59.055 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T07:43:00.091 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T07:43:00.130 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 12/12 2024-04-16T07:43:02.647 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 12/12 2024-04-16T07:43:02.647 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 1/12 2024-04-16T07:43:02.648 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T07:43:02.648 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:43:02.648 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T07:43:02.648 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:43:02.648 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:43:02.648 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T07:43:02.648 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T07:43:02.648 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T07:43:02.648 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T07:43:02.648 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T07:43:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T07:43:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:03.137 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-16T07:43:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:43:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T07:43:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T07:43:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T07:43:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T07:43:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T07:43:03.138 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T07:43:03.138 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T07:43:03.138 INFO:teuthology.orchestra.run.smithi172.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T07:43:03.138 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T07:43:03.138 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T07:43:03.138 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T07:43:03.138 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:03.138 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:43:03.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-rook 2024-04-16T07:43:04.149 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:08 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:43:04.302 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:43:04.304 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2024-04-16T07:43:04.304 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-16T07:43:04.304 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2024-04-16T07:43:04.304 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-16T07:43:04.304 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 49 k 2024-04-16T07:43:04.304 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-16T07:43:04.304 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T07:43:04.304 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T07:43:04.304 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T07:43:04.304 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T07:43:04.304 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T07:43:04.304 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T07:43:04.305 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T07:43:04.305 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T07:43:04.305 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T07:43:04.305 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T07:43:04.305 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T07:43:04.305 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T07:43:04.305 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:04.305 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-16T07:43:04.305 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2024-04-16T07:43:04.306 INFO:teuthology.orchestra.run.smithi172.stdout:Install 13 Packages 2024-04-16T07:43:04.306 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:04.306 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 2.0 M 2024-04-16T07:43:04.306 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 26 M 2024-04-16T07:43:04.307 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-16T07:43:04.646 INFO:teuthology.orchestra.run.smithi172.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 104 kB/s | 18 kB 00:00 2024-04-16T07:43:04.737 INFO:teuthology.orchestra.run.smithi172.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 92 kB/s | 24 kB 00:00 2024-04-16T07:43:04.787 INFO:teuthology.orchestra.run.smithi172.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 174 kB 00:00 2024-04-16T07:43:04.812 INFO:teuthology.orchestra.run.smithi172.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-16T07:43:04.838 INFO:teuthology.orchestra.run.smithi172.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-04-16T07:43:04.863 INFO:teuthology.orchestra.run.smithi172.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-04-16T07:43:04.997 INFO:teuthology.orchestra.run.smithi172.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 112 kB/s | 15 kB 00:00 2024-04-16T07:43:05.022 INFO:teuthology.orchestra.run.smithi172.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 175 kB/s | 32 kB 00:00 2024-04-16T07:43:05.247 INFO:teuthology.orchestra.run.smithi172.stdout:(9/13): ceph-mgr-rook-19.0.0-3109.g30b0aa00.el9 63 kB/s | 49 kB 00:00 2024-04-16T07:43:05.306 INFO:teuthology.orchestra.run.smithi172.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 656 kB/s | 203 kB 00:00 2024-04-16T07:43:05.415 INFO:teuthology.orchestra.run.smithi172.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 828 kB/s | 90 kB 00:00 2024-04-16T07:43:05.540 INFO:teuthology.orchestra.run.smithi172.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 202 kB/s | 59 kB 00:00 2024-04-16T07:43:05.665 INFO:teuthology.orchestra.run.smithi172.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.6 MB/s | 1.0 MB 00:00 2024-04-16T07:43:05.667 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:05.667 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-04-16T07:43:05.668 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-16T07:43:05.682 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-16T07:43:05.683 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-16T07:43:05.780 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-16T07:43:05.782 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-16T07:43:06.298 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-16T07:43:06.461 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T07:43:06.524 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T07:43:06.588 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T07:43:06.625 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T07:43:06.667 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T07:43:06.761 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T07:43:06.862 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T07:43:06.918 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T07:43:07.637 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T07:43:07.690 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T07:43:07.724 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T07:43:07.772 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T07:43:07.802 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 13/13 2024-04-16T07:43:08.583 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 13/13 2024-04-16T07:43:08.583 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 1/13 2024-04-16T07:43:08.583 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T07:43:08.583 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T07:43:08.583 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T07:43:08.583 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T07:43:08.583 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T07:43:08.583 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T07:43:08.583 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T07:43:08.583 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T07:43:08.583 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T07:43:08.583 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T07:43:08.584 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T07:43:08.937 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T07:43:08.937 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:08.937 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-16T07:43:08.937 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:43:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T07:43:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T07:43:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T07:43:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T07:43:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T07:43:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T07:43:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T07:43:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T07:43:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T07:43:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T07:43:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T07:43:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T07:43:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:08.939 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:43:09.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T07:43:09.777 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:13 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:43:09.931 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:43:09.931 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2024-04-16T07:43:09.931 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-16T07:43:09.932 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2024-04-16T07:43:09.932 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-16T07:43:09.932 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 142 k 2024-04-16T07:43:09.932 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-16T07:43:09.932 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T07:43:09.932 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T07:43:09.932 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T07:43:09.932 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T07:43:09.932 INFO:teuthology.orchestra.run.smithi172.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T07:43:09.932 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:09.932 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-16T07:43:09.932 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2024-04-16T07:43:09.932 INFO:teuthology.orchestra.run.smithi172.stdout:Install 6 Packages 2024-04-16T07:43:09.932 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:09.933 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 6.8 M 2024-04-16T07:43:09.933 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 33 M 2024-04-16T07:43:09.933 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-16T07:43:10.669 INFO:teuthology.orchestra.run.smithi172.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-04-16T07:43:10.983 INFO:teuthology.orchestra.run.smithi172.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-04-16T07:43:11.042 INFO:teuthology.orchestra.run.smithi172.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 989 kB/s | 58 kB 00:00 2024-04-16T07:43:11.100 INFO:teuthology.orchestra.run.smithi172.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 875 kB/s | 51 kB 00:00 2024-04-16T07:43:11.333 INFO:teuthology.orchestra.run.smithi172.stdout:(5/6): ceph-mgr-cephadm-19.0.0-3109.g30b0aa00.e 145 kB/s | 142 kB 00:00 2024-04-16T07:43:11.384 INFO:teuthology.orchestra.run.smithi172.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 240 kB/s | 248 kB 00:01 2024-04-16T07:43:11.386 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:11.386 INFO:teuthology.orchestra.run.smithi172.stdout:Total 4.7 MB/s | 6.8 MB 00:01 2024-04-16T07:43:11.399 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-16T07:43:11.410 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-16T07:43:11.410 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-16T07:43:11.500 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-16T07:43:11.501 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-16T07:43:11.805 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-16T07:43:11.973 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T07:43:12.018 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T07:43:12.531 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T07:43:12.620 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T07:43:12.683 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T07:43:12.718 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 6/6 2024-04-16T07:43:13.520 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 6/6 2024-04-16T07:43:13.520 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/6 2024-04-16T07:43:13.520 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T07:43:13.520 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T07:43:13.520 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T07:43:13.520 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T07:43:13.815 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T07:43:13.815 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:13.815 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-16T07:43:13.816 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:43:13.816 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T07:43:13.816 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T07:43:13.816 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T07:43:13.816 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T07:43:13.816 INFO:teuthology.orchestra.run.smithi172.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T07:43:13.816 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:13.816 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:43:14.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-fuse 2024-04-16T07:43:14.695 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:18 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:43:14.848 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:43:14.848 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:14.848 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-16T07:43:14.848 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:14.848 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-16T07:43:14.848 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 871 k 2024-04-16T07:43:14.848 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-16T07:43:14.848 INFO:teuthology.orchestra.run.smithi172.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T07:43:14.849 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:14.849 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-16T07:43:14.849 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:14.849 INFO:teuthology.orchestra.run.smithi172.stdout:Install 2 Packages 2024-04-16T07:43:14.849 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:14.849 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 950 k 2024-04-16T07:43:14.849 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 4.0 M 2024-04-16T07:43:14.850 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-16T07:43:14.962 INFO:teuthology.orchestra.run.smithi172.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 704 kB/s | 79 kB 00:00 2024-04-16T07:43:18.603 INFO:teuthology.orchestra.run.smithi172.stdout:(2/2): ceph-fuse-19.0.0-3109.g30b0aa00.el9.x86_ 232 kB/s | 871 kB 00:03 2024-04-16T07:43:18.605 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:18.605 INFO:teuthology.orchestra.run.smithi172.stdout:Total 253 kB/s | 950 kB 00:03 2024-04-16T07:43:18.605 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-16T07:43:18.631 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-16T07:43:18.631 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-16T07:43:18.707 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-16T07:43:18.707 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-16T07:43:18.884 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-16T07:43:19.046 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T07:43:19.120 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/2 2024-04-16T07:43:19.822 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/2 2024-04-16T07:43:19.822 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/2 2024-04-16T07:43:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T07:43:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:20.149 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-16T07:43:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T07:43:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:20.149 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:43:20.397 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-volume 2024-04-16T07:43:20.831 INFO:teuthology.orchestra.run.smithi070.stdout:(36/36): ceph-osd-19.0.0-3109.g30b0aa00.el9.x86 565 kB/s | 71 MB 02:08 2024-04-16T07:43:20.834 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:20.834 INFO:teuthology.orchestra.run.smithi070.stdout:Total 655 kB/s | 84 MB 02:11 2024-04-16T07:43:20.853 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-16T07:43:20.891 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-16T07:43:20.891 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-16T07:43:20.996 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:24 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:43:21.154 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:43:21.155 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:21.155 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-16T07:43:21.155 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:21.155 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-16T07:43:21.155 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 273 k 2024-04-16T07:43:21.155 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-16T07:43:21.155 INFO:teuthology.orchestra.run.smithi172.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-16T07:43:21.155 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:21.155 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-16T07:43:21.155 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:21.156 INFO:teuthology.orchestra.run.smithi172.stdout:Install 2 Packages 2024-04-16T07:43:21.156 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:21.156 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 342 k 2024-04-16T07:43:21.156 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 1.5 M 2024-04-16T07:43:21.156 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-16T07:43:21.265 INFO:teuthology.orchestra.run.smithi172.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 633 kB/s | 69 kB 00:00 2024-04-16T07:43:21.627 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-16T07:43:21.627 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-16T07:43:23.221 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-16T07:43:23.283 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-16T07:43:23.318 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-16T07:43:23.344 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-16T07:43:23.414 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-16T07:43:23.451 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-16T07:43:23.484 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-16T07:43:23.575 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-16T07:43:23.662 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-16T07:43:23.703 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-16T07:43:23.775 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-16T07:43:23.829 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-16T07:43:23.867 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-16T07:43:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-16T07:43:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-16T07:43:24.159 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-16T07:43:24.205 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-16T07:43:24.241 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-16T07:43:24.286 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-16T07:43:24.316 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-16T07:43:24.393 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-16T07:43:24.427 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T07:43:24.460 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-16T07:43:24.501 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-16T07:43:24.536 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-16T07:43:24.570 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-16T07:43:24.661 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-16T07:43:24.709 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-16T07:43:24.781 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-16T07:43:24.918 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-16T07:43:25.045 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T07:43:25.246 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9. 31/36 2024-04-16T07:43:25.323 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/36 2024-04-16T07:43:25.348 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/36 2024-04-16T07:43:25.348 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:43:25.348 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T07:43:25.348 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:43:25.349 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:43:25.349 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:43:25.648 INFO:teuthology.orchestra.run.smithi172.stdout:(2/2): ceph-volume-19.0.0-3109.g30b0aa00.el9.no 61 kB/s | 273 kB 00:04 2024-04-16T07:43:25.650 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:25.651 INFO:teuthology.orchestra.run.smithi172.stdout:Total 76 kB/s | 342 kB 00:04 2024-04-16T07:43:25.651 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-16T07:43:25.661 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-16T07:43:25.661 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-16T07:43:25.709 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-16T07:43:25.710 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-16T07:43:25.943 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-16T07:43:26.114 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-16T07:43:26.143 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2/2 2024-04-16T07:43:26.159 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2/2 2024-04-16T07:43:26.159 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:43:26.159 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-16T07:43:26.159 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:26.895 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 1/2 2024-04-16T07:43:27.213 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-16T07:43:27.213 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:27.214 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-16T07:43:27.214 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:43:27.214 INFO:teuthology.orchestra.run.smithi172.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-16T07:43:27.214 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:27.214 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:43:27.537 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librados-devel 2024-04-16T07:43:28.137 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:32 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:43:28.289 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:43:28.290 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:28.290 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-04-16T07:43:28.290 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:28.290 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-16T07:43:28.290 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 125 k 2024-04-16T07:43:28.290 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:28.291 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-16T07:43:28.291 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:28.291 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-04-16T07:43:28.291 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:28.291 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 125 k 2024-04-16T07:43:28.291 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 445 k 2024-04-16T07:43:28.291 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-16T07:43:28.970 INFO:teuthology.orchestra.run.smithi172.stdout:librados-devel-19.0.0-3109.g30b0aa00.el9.x86_64 184 kB/s | 125 kB 00:00 2024-04-16T07:43:28.971 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:28.971 INFO:teuthology.orchestra.run.smithi172.stdout:Total 183 kB/s | 125 kB 00:00 2024-04-16T07:43:28.971 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-16T07:43:28.977 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-16T07:43:28.977 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-16T07:43:29.032 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-16T07:43:29.033 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-16T07:43:29.185 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-16T07:43:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:29.740 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:30.029 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:30.030 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:30.030 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-16T07:43:30.030 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:30.030 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:30.030 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:43:30.297 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs2 2024-04-16T07:43:30.903 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:34 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:43:31.009 INFO:teuthology.orchestra.run.smithi172.stdout:Package libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:43:31.065 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:43:31.066 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-16T07:43:31.066 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:43:31.134 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs-devel 2024-04-16T07:43:31.757 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:35 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:43:31.916 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:43:31.917 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:31.917 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-04-16T07:43:31.917 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:31.918 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-16T07:43:31.918 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 32 k 2024-04-16T07:43:31.918 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:31.918 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-16T07:43:31.918 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:31.918 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-04-16T07:43:31.918 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:31.918 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 32 k 2024-04-16T07:43:31.918 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 142 k 2024-04-16T07:43:31.918 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-16T07:43:32.457 INFO:teuthology.orchestra.run.smithi172.stdout:libcephfs-devel-19.0.0-3109.g30b0aa00.el9.x86_6 59 kB/s | 32 kB 00:00 2024-04-16T07:43:32.458 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:32.458 INFO:teuthology.orchestra.run.smithi172.stdout:Total 59 kB/s | 32 kB 00:00 2024-04-16T07:43:32.458 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-16T07:43:32.462 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-16T07:43:32.462 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-16T07:43:32.490 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-16T07:43:32.491 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-16T07:43:32.578 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-16T07:43:32.706 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:33.065 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:33.323 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:33.324 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:33.324 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-16T07:43:33.324 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:33.324 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:33.324 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:43:33.533 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librados2 2024-04-16T07:43:34.119 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:38 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:43:34.219 INFO:teuthology.orchestra.run.smithi172.stdout:Package librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:43:34.272 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:43:34.274 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-16T07:43:34.274 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:43:34.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librbd1 2024-04-16T07:43:34.936 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:38 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:43:35.036 INFO:teuthology.orchestra.run.smithi172.stdout:Package librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:43:35.088 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:43:35.089 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-16T07:43:35.089 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:43:35.167 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rados 2024-04-16T07:43:35.754 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:39 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:43:35.854 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:43:35.907 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:43:35.908 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-16T07:43:35.908 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:43:36.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rgw 2024-04-16T07:43:36.097 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/36 2024-04-16T07:43:36.124 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/36 2024-04-16T07:43:36.124 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:43:36.124 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T07:43:36.124 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:43:36.124 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:43:36.124 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:43:36.585 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:40 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:43:36.664 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 34/36 2024-04-16T07:43:36.685 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:43:36.687 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 34/36 2024-04-16T07:43:36.687 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:43:36.687 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T07:43:36.687 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:43:36.687 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:43:36.687 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:43:36.737 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:43:36.738 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-16T07:43:36.738 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:43:36.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-cephfs 2024-04-16T07:43:36.923 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 35/36 2024-04-16T07:43:36.946 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 35/36 2024-04-16T07:43:36.946 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:43:36.946 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T07:43:36.947 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:43:36.947 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:43:36.947 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:43:37.012 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 36/36 2024-04-16T07:43:37.395 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:41 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:43:37.494 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:43:37.545 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:43:37.547 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-16T07:43:37.547 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:43:37.617 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rbd 2024-04-16T07:43:38.204 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:42 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:43:38.305 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:43:38.357 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:43:38.358 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-16T07:43:38.358 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:43:38.421 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-fuse 2024-04-16T07:43:39.005 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:43 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:43:39.158 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:43:39.158 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:39.158 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-04-16T07:43:39.158 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:39.158 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-16T07:43:39.158 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 90 k 2024-04-16T07:43:39.158 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:39.159 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-16T07:43:39.159 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:39.159 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-04-16T07:43:39.159 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:39.159 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 90 k 2024-04-16T07:43:39.160 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 313 k 2024-04-16T07:43:39.160 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-16T07:43:40.918 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-fuse-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 51 kB/s | 90 kB 00:01 2024-04-16T07:43:40.920 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:40.920 INFO:teuthology.orchestra.run.smithi172.stdout:Total 51 kB/s | 90 kB 00:01 2024-04-16T07:43:40.920 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-16T07:43:40.930 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-16T07:43:40.930 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-16T07:43:40.991 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-16T07:43:40.991 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-16T07:43:41.138 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-16T07:43:41.243 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:41.624 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:41.913 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:41.913 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:41.913 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-16T07:43:41.913 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:41.913 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:41.913 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:43:42.131 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-mirror 2024-04-16T07:43:42.727 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:46 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:43:42.883 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:43:42.883 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:42.883 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-04-16T07:43:42.884 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:42.884 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-16T07:43:42.884 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.4 M 2024-04-16T07:43:42.884 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:42.884 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-16T07:43:42.884 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:42.884 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-04-16T07:43:42.884 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:42.885 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 3.4 M 2024-04-16T07:43:42.885 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 29 M 2024-04-16T07:43:42.885 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-16T07:43:43.548 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 36/36 2024-04-16T07:43:43.548 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/36 2024-04-16T07:43:43.548 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/36 2024-04-16T07:43:43.548 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/36 2024-04-16T07:43:43.548 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/36 2024-04-16T07:43:43.548 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 5/36 2024-04-16T07:43:43.548 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9. 6/36 2024-04-16T07:43:43.548 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-16T07:43:43.548 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-16T07:43:43.548 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-16T07:43:43.548 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-16T07:43:43.549 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-16T07:43:43.549 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-16T07:43:43.549 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-16T07:43:43.549 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-16T07:43:43.549 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-16T07:43:43.549 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-16T07:43:43.549 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-16T07:43:43.549 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-16T07:43:43.549 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-16T07:43:43.549 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-16T07:43:43.549 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T07:43:43.549 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-16T07:43:43.549 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-16T07:43:43.549 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-16T07:43:43.550 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-16T07:43:43.551 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-16T07:43:43.551 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-16T07:43:43.551 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-16T07:43:43.551 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-16T07:43:43.551 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T07:43:43.551 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-16T07:43:43.551 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-16T07:43:43.551 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-16T07:43:43.551 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-16T07:43:43.551 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-16T07:43:44.570 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-16T07:43:44.570 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:43:44.570 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-16T07:43:44.570 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:44.570 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:44.570 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:44.570 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:43:44.570 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:44.570 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:44.570 INFO:teuthology.orchestra.run.smithi070.stdout: lua-5.4.4-4.el9.x86_64 2024-04-16T07:43:44.570 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-16T07:43:44.570 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-16T07:43:44.571 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T07:43:44.571 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T07:43:44.571 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T07:43:44.571 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T07:43:44.571 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T07:43:44.571 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T07:43:44.571 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T07:43:44.571 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T07:43:44.571 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T07:43:44.571 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T07:43:44.571 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T07:43:44.571 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T07:43:44.571 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T07:43:44.571 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T07:43:44.571 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T07:43:44.571 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T07:43:44.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T07:43:44.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T07:43:44.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T07:43:44.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T07:43:44.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T07:43:44.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T07:43:44.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T07:43:44.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T07:43:44.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T07:43:44.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T07:43:44.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T07:43:44.572 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:43:44.572 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:43:44.833 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-base 2024-04-16T07:43:45.413 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:06:06 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:43:45.511 INFO:teuthology.orchestra.run.smithi070.stdout:Package ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:43:45.562 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:43:45.563 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-04-16T07:43:45.563 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:43:45.619 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install cephadm 2024-04-16T07:43:46.193 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:06:07 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:43:46.343 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:43:46.344 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:43:46.344 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-04-16T07:43:46.344 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:43:46.344 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-16T07:43:46.344 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 762 k 2024-04-16T07:43:46.344 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:43:46.345 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-16T07:43:46.345 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:43:46.345 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-04-16T07:43:46.345 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:43:46.345 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 762 k 2024-04-16T07:43:46.345 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 769 k 2024-04-16T07:43:46.345 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-16T07:43:48.109 INFO:teuthology.orchestra.run.smithi070.stdout:cephadm-19.0.0-3109.g30b0aa00.el9.noarch.rpm 432 kB/s | 762 kB 00:01 2024-04-16T07:43:48.110 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:48.110 INFO:teuthology.orchestra.run.smithi070.stdout:Total 432 kB/s | 762 kB 00:01 2024-04-16T07:43:48.110 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-16T07:43:48.117 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-16T07:43:48.117 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-16T07:43:48.123 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-16T07:43:48.123 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-16T07:43:48.196 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-16T07:43:48.675 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:43:48.756 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:43:49.113 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:43:49.414 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:43:49.414 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:43:49.414 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-16T07:43:49.414 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:43:49.414 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:43:49.414 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:43:49.420 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-mirror-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 533 kB/s | 3.4 MB 00:06 2024-04-16T07:43:49.421 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:49.422 INFO:teuthology.orchestra.run.smithi172.stdout:Total 533 kB/s | 3.4 MB 00:06 2024-04-16T07:43:49.422 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-16T07:43:49.435 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-16T07:43:49.435 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-16T07:43:49.514 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-16T07:43:49.515 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-16T07:43:49.617 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T07:43:50.073 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-16T07:43:50.137 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:50.163 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:50.163 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:43:50.163 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T07:43:50.163 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:43:50.163 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:43:50.163 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:50.210 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:06:11 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:43:50.359 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:43:50.360 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:43:50.360 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-04-16T07:43:50.360 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:43:50.360 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-16T07:43:50.360 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 157 k 2024-04-16T07:43:50.360 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:43:50.360 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-16T07:43:50.360 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:43:50.361 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-04-16T07:43:50.361 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:43:50.361 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 157 k 2024-04-16T07:43:50.361 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 640 k 2024-04-16T07:43:50.361 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-16T07:43:51.182 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-immutable-object-cache-19.0.0-3109.g30b0aa 192 kB/s | 157 kB 00:00 2024-04-16T07:43:51.183 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:51.183 INFO:teuthology.orchestra.run.smithi070.stdout:Total 191 kB/s | 157 kB 00:00 2024-04-16T07:43:51.183 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-16T07:43:51.193 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-16T07:43:51.193 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-16T07:43:51.246 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-16T07:43:51.246 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-16T07:43:51.385 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-16T07:43:51.450 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:43:51.467 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:51.467 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:51.468 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-16T07:43:51.468 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:51.468 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:51.468 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:43:51.474 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:43:51.475 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:43:51.475 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T07:43:51.475 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:43:51.722 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-nbd 2024-04-16T07:43:52.322 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:56 ago on Tue 16 Apr 2024 07:37:56 AM UTC. 2024-04-16T07:43:52.475 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-16T07:43:52.475 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:52.475 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-04-16T07:43:52.476 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:52.476 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-16T07:43:52.476 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 175 k 2024-04-16T07:43:52.476 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:52.476 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-16T07:43:52.476 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-16T07:43:52.476 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-04-16T07:43:52.476 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:52.477 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:43:52.477 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:43:52.477 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-16T07:43:52.477 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:52.477 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:43:52.477 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:43:52.477 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 175 k 2024-04-16T07:43:52.477 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 673 k 2024-04-16T07:43:52.477 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-16T07:43:52.703 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr 2024-04-16T07:43:53.285 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:06:14 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:43:53.382 INFO:teuthology.orchestra.run.smithi070.stdout:Package ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:43:53.436 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:43:53.437 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-04-16T07:43:53.437 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:43:53.504 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T07:43:53.590 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-nbd-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 158 kB/s | 175 kB 00:01 2024-04-16T07:43:53.591 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:53.591 INFO:teuthology.orchestra.run.smithi172.stdout:Total 157 kB/s | 175 kB 00:01 2024-04-16T07:43:53.591 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-16T07:43:53.603 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-16T07:43:53.603 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-16T07:43:53.663 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-16T07:43:53.663 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-16T07:43:53.840 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-16T07:43:53.939 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:54.074 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:06:15 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:43:54.226 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:43:54.228 INFO:teuthology.orchestra.run.smithi070.stdout:====================================================================================== 2024-04-16T07:43:54.228 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-04-16T07:43:54.228 INFO:teuthology.orchestra.run.smithi070.stdout:====================================================================================== 2024-04-16T07:43:54.228 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-16T07:43:54.228 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 3.5 M 2024-04-16T07:43:54.228 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-04-16T07:43:54.228 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-16T07:43:54.228 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 34 k 2024-04-16T07:43:54.228 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 16 k 2024-04-16T07:43:54.228 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-16T07:43:54.228 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-16T07:43:54.229 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-16T07:43:54.229 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-16T07:43:54.229 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-16T07:43:54.229 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T07:43:54.229 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T07:43:54.229 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:43:54.229 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-16T07:43:54.229 INFO:teuthology.orchestra.run.smithi070.stdout:====================================================================================== 2024-04-16T07:43:54.229 INFO:teuthology.orchestra.run.smithi070.stdout:Install 11 Packages 2024-04-16T07:43:54.229 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:43:54.230 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 7.6 M 2024-04-16T07:43:54.230 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 100 M 2024-04-16T07:43:54.230 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-16T07:43:54.365 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:54.667 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:54.667 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:54.667 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-16T07:43:54.668 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:54.668 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:43:54.668 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-16T07:43:54.861 INFO:teuthology.orchestra.run.smithi070.stdout:(1/11): ceph-grafana-dashboards-19.0.0-3109.g30 83 kB/s | 34 kB 00:00 2024-04-16T07:43:54.903 DEBUG:teuthology.parallel:result is None 2024-04-16T07:43:54.927 INFO:teuthology.orchestra.run.smithi070.stdout:(2/11): ceph-prometheus-alerts-19.0.0-3109.g30b 34 kB/s | 16 kB 00:00 2024-04-16T07:43:55.028 INFO:teuthology.orchestra.run.smithi070.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 238 kB 00:00 2024-04-16T07:43:55.370 INFO:teuthology.orchestra.run.smithi070.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.6 MB/s | 548 kB 00:00 2024-04-16T07:43:55.412 INFO:teuthology.orchestra.run.smithi070.stdout:(5/11): protobuf-compiler-3.14.0-13.el9.x86_64. 1.7 MB/s | 863 kB 00:00 2024-04-16T07:43:55.429 INFO:teuthology.orchestra.run.smithi070.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 332 kB/s | 19 kB 00:00 2024-04-16T07:43:55.487 INFO:teuthology.orchestra.run.smithi070.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-04-16T07:43:55.538 INFO:teuthology.orchestra.run.smithi070.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 616 kB/s | 31 kB 00:00 2024-04-16T07:43:55.604 INFO:teuthology.orchestra.run.smithi070.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-04-16T07:43:55.830 INFO:teuthology.orchestra.run.smithi070.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.9 MB/s | 2.0 MB 00:00 2024-04-16T07:43:59.996 INFO:teuthology.orchestra.run.smithi070.stdout:(11/11): ceph-mgr-dashboard-19.0.0-3109.g30b0aa 654 kB/s | 3.5 MB 00:05 2024-04-16T07:43:59.999 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:59.999 INFO:teuthology.orchestra.run.smithi070.stdout:Total 1.3 MB/s | 7.6 MB 00:05 2024-04-16T07:44:00.020 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-16T07:44:00.038 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-16T07:44:00.038 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-16T07:44:00.215 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-16T07:44:00.216 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-16T07:44:00.692 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-16T07:44:00.769 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-16T07:44:00.799 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-16T07:44:00.914 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-16T07:44:01.049 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-16T07:44:01.094 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-16T07:44:01.177 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-16T07:44:01.221 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-16T07:44:01.250 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-16T07:44:01.282 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9 9/11 2024-04-16T07:44:02.321 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el 10/11 2024-04-16T07:44:02.374 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 11/11 2024-04-16T07:44:03.690 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 11/11 2024-04-16T07:44:03.690 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el 1/11 2024-04-16T07:44:03.690 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 2/11 2024-04-16T07:44:03.690 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9 3/11 2024-04-16T07:44:03.690 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-16T07:44:03.690 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-16T07:44:03.691 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-16T07:44:03.691 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-16T07:44:03.691 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-16T07:44:03.691 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-16T07:44:03.691 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-16T07:44:04.039 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-16T07:44:04.039 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:04.039 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-16T07:44:04.039 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-16T07:44:04.039 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:44:04.039 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:44:04.039 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:44:04.039 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-16T07:44:04.039 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-16T07:44:04.039 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-16T07:44:04.039 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-16T07:44:04.039 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-16T07:44:04.039 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T07:44:04.040 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T07:44:04.040 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:04.040 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:44:04.285 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T07:44:04.862 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:06:25 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:44:05.014 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:44:05.015 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================================ 2024-04-16T07:44:05.015 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-04-16T07:44:05.015 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================================ 2024-04-16T07:44:05.016 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-16T07:44:05.016 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 7.4 M 2024-04-16T07:44:05.016 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-04-16T07:44:05.016 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T07:44:05.016 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T07:44:05.016 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T07:44:05.016 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T07:44:05.016 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T07:44:05.016 INFO:teuthology.orchestra.run.smithi070.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T07:44:05.016 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T07:44:05.016 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T07:44:05.016 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T07:44:05.016 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T07:44:05.016 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T07:44:05.017 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:05.017 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-16T07:44:05.017 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================================ 2024-04-16T07:44:05.017 INFO:teuthology.orchestra.run.smithi070.stdout:Install 12 Packages 2024-04-16T07:44:05.017 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:05.017 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 37 M 2024-04-16T07:44:05.018 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 209 M 2024-04-16T07:44:05.018 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-16T07:44:05.313 INFO:teuthology.orchestra.run.smithi070.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 300 kB/s | 32 kB 00:00 2024-04-16T07:44:05.363 INFO:teuthology.orchestra.run.smithi070.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 634 kB/s | 17 kB 00:00 2024-04-16T07:44:05.396 INFO:teuthology.orchestra.run.smithi070.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-04-16T07:44:05.597 INFO:teuthology.orchestra.run.smithi070.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2024-04-16T07:44:05.747 INFO:teuthology.orchestra.run.smithi070.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-04-16T07:44:05.789 INFO:teuthology.orchestra.run.smithi070.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-04-16T07:44:06.189 INFO:teuthology.orchestra.run.smithi070.stdout:(7/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-04-16T07:44:06.239 INFO:teuthology.orchestra.run.smithi070.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.9 MB/s | 3.0 MB 00:01 2024-04-16T07:44:07.232 INFO:teuthology.orchestra.run.smithi070.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 197 kB/s | 195 kB 00:00 2024-04-16T07:44:07.373 INFO:teuthology.orchestra.run.smithi070.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-04-16T07:44:07.957 INFO:teuthology.orchestra.run.smithi070.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 454 kB/s | 803 kB 00:01 2024-04-16T07:44:15.247 INFO:teuthology.orchestra.run.smithi070.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-3 754 kB/s | 7.4 MB 00:10 2024-04-16T07:44:15.251 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:44:15.251 INFO:teuthology.orchestra.run.smithi070.stdout:Total 3.6 MB/s | 37 MB 00:10 2024-04-16T07:44:15.296 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-16T07:44:15.321 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-16T07:44:15.321 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-16T07:44:15.674 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-16T07:44:15.675 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-16T07:44:16.371 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-16T07:44:16.445 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T07:44:16.476 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T07:44:16.577 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:44:16.613 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T07:44:16.789 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:44:16.821 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:44:16.896 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T07:44:17.544 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T07:44:17.665 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T07:44:18.858 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T07:44:19.882 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T07:44:19.921 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 12/12 2024-04-16T07:44:22.463 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 12/12 2024-04-16T07:44:22.463 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 1/12 2024-04-16T07:44:22.463 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T07:44:22.463 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:44:22.463 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T07:44:22.463 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:44:22.463 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:44:22.463 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T07:44:22.463 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T07:44:22.463 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T07:44:22.463 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T07:44:22.464 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T07:44:22.894 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T07:44:22.894 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:22.894 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-16T07:44:22.894 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:44:22.894 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T07:44:22.894 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T07:44:22.894 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T07:44:22.894 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T07:44:22.894 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T07:44:22.895 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T07:44:22.895 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T07:44:22.895 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T07:44:22.895 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T07:44:22.895 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T07:44:22.895 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T07:44:22.895 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:22.895 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:44:23.304 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-rook 2024-04-16T07:44:23.886 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:06:44 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:44:24.038 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:44:24.040 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================== 2024-04-16T07:44:24.040 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-04-16T07:44:24.040 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================== 2024-04-16T07:44:24.040 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-16T07:44:24.040 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 49 k 2024-04-16T07:44:24.040 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-04-16T07:44:24.040 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T07:44:24.040 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T07:44:24.041 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T07:44:24.041 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T07:44:24.041 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T07:44:24.041 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T07:44:24.041 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T07:44:24.041 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T07:44:24.041 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T07:44:24.041 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T07:44:24.041 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T07:44:24.041 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T07:44:24.041 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:24.041 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-16T07:44:24.041 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================== 2024-04-16T07:44:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:Install 13 Packages 2024-04-16T07:44:24.042 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 2.0 M 2024-04-16T07:44:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 26 M 2024-04-16T07:44:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-16T07:44:24.228 INFO:teuthology.orchestra.run.smithi070.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 238 kB/s | 24 kB 00:00 2024-04-16T07:44:24.279 INFO:teuthology.orchestra.run.smithi070.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 118 kB/s | 18 kB 00:00 2024-04-16T07:44:24.320 INFO:teuthology.orchestra.run.smithi070.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-04-16T07:44:24.346 INFO:teuthology.orchestra.run.smithi070.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-04-16T07:44:24.371 INFO:teuthology.orchestra.run.smithi070.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-04-16T07:44:24.396 INFO:teuthology.orchestra.run.smithi070.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-04-16T07:44:24.772 INFO:teuthology.orchestra.run.smithi070.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 40 kB/s | 15 kB 00:00 2024-04-16T07:44:24.830 INFO:teuthology.orchestra.run.smithi070.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 70 kB/s | 32 kB 00:00 2024-04-16T07:44:25.072 INFO:teuthology.orchestra.run.smithi070.stdout:(9/13): ceph-mgr-rook-19.0.0-3109.g30b0aa00.el9 52 kB/s | 49 kB 00:00 2024-04-16T07:44:25.172 INFO:teuthology.orchestra.run.smithi070.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 506 kB/s | 203 kB 00:00 2024-04-16T07:44:25.364 INFO:teuthology.orchestra.run.smithi070.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 538 kB/s | 90 kB 00:00 2024-04-16T07:44:25.448 INFO:teuthology.orchestra.run.smithi070.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-04-16T07:44:25.681 INFO:teuthology.orchestra.run.smithi070.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 97 kB/s | 59 kB 00:00 2024-04-16T07:44:25.687 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:44:25.687 INFO:teuthology.orchestra.run.smithi070.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-04-16T07:44:25.687 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-16T07:44:25.702 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-16T07:44:25.702 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-16T07:44:25.799 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-16T07:44:25.799 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-16T07:44:26.316 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-16T07:44:26.478 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T07:44:26.541 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T07:44:26.604 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T07:44:26.642 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T07:44:26.683 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T07:44:26.776 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T07:44:26.877 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T07:44:26.926 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T07:44:27.648 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T07:44:27.690 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T07:44:27.724 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T07:44:27.772 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T07:44:27.794 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 13/13 2024-04-16T07:44:28.539 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 13/13 2024-04-16T07:44:28.539 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 1/13 2024-04-16T07:44:28.539 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T07:44:28.539 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T07:44:28.539 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T07:44:28.539 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T07:44:28.539 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T07:44:28.539 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T07:44:28.539 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T07:44:28.539 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T07:44:28.539 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T07:44:28.539 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T07:44:28.540 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T07:44:28.905 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T07:44:28.905 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:28.905 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-16T07:44:28.905 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:44:28.905 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T07:44:28.906 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T07:44:28.906 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T07:44:28.906 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T07:44:28.906 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T07:44:28.906 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T07:44:28.906 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T07:44:28.906 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T07:44:28.906 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T07:44:28.906 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T07:44:28.906 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T07:44:28.906 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T07:44:28.906 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:28.906 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:44:29.166 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T07:44:29.741 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:06:50 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:44:29.894 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:44:29.895 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================== 2024-04-16T07:44:29.895 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-04-16T07:44:29.895 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================== 2024-04-16T07:44:29.895 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-16T07:44:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 142 k 2024-04-16T07:44:29.896 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-04-16T07:44:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T07:44:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T07:44:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T07:44:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T07:44:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T07:44:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:29.896 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-16T07:44:29.896 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================== 2024-04-16T07:44:29.896 INFO:teuthology.orchestra.run.smithi070.stdout:Install 6 Packages 2024-04-16T07:44:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:29.897 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 6.8 M 2024-04-16T07:44:29.897 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 33 M 2024-04-16T07:44:29.897 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-16T07:44:30.658 INFO:teuthology.orchestra.run.smithi070.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.5 MB/s | 248 kB 00:00 2024-04-16T07:44:30.850 INFO:teuthology.orchestra.run.smithi070.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-04-16T07:44:31.000 INFO:teuthology.orchestra.run.smithi070.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 387 kB/s | 58 kB 00:00 2024-04-16T07:44:31.025 INFO:teuthology.orchestra.run.smithi070.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-04-16T07:44:31.067 INFO:teuthology.orchestra.run.smithi070.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 763 kB/s | 51 kB 00:00 2024-04-16T07:44:31.743 INFO:teuthology.orchestra.run.smithi070.stdout:(6/6): ceph-mgr-cephadm-19.0.0-3109.g30b0aa00.e 120 kB/s | 142 kB 00:01 2024-04-16T07:44:31.744 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:44:31.745 INFO:teuthology.orchestra.run.smithi070.stdout:Total 3.7 MB/s | 6.8 MB 00:01 2024-04-16T07:44:31.760 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-16T07:44:31.771 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-16T07:44:31.771 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-16T07:44:31.861 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-16T07:44:31.861 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-16T07:44:32.174 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-16T07:44:32.347 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T07:44:32.394 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T07:44:32.902 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T07:44:32.988 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T07:44:33.052 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T07:44:33.081 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 6/6 2024-04-16T07:44:33.874 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 6/6 2024-04-16T07:44:33.874 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/6 2024-04-16T07:44:33.874 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T07:44:33.874 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T07:44:33.874 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T07:44:33.874 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T07:44:34.205 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T07:44:34.206 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:34.206 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-16T07:44:34.206 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:44:34.206 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T07:44:34.206 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T07:44:34.206 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T07:44:34.206 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T07:44:34.206 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T07:44:34.206 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:34.206 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:44:34.476 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-fuse 2024-04-16T07:44:35.054 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:06:56 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:44:35.207 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:44:35.208 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:44:35.208 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-04-16T07:44:35.208 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:44:35.208 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-16T07:44:35.208 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 871 k 2024-04-16T07:44:35.208 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-04-16T07:44:35.208 INFO:teuthology.orchestra.run.smithi070.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T07:44:35.208 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:35.209 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-16T07:44:35.209 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:44:35.209 INFO:teuthology.orchestra.run.smithi070.stdout:Install 2 Packages 2024-04-16T07:44:35.209 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:35.209 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 950 k 2024-04-16T07:44:35.209 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 4.0 M 2024-04-16T07:44:35.209 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-16T07:44:35.370 INFO:teuthology.orchestra.run.smithi070.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 490 kB/s | 79 kB 00:00 2024-04-16T07:44:36.920 INFO:teuthology.orchestra.run.smithi070.stdout:(2/2): ceph-fuse-19.0.0-3109.g30b0aa00.el9.x86_ 509 kB/s | 871 kB 00:01 2024-04-16T07:44:36.922 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:44:36.922 INFO:teuthology.orchestra.run.smithi070.stdout:Total 554 kB/s | 950 kB 00:01 2024-04-16T07:44:36.922 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-16T07:44:36.947 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-16T07:44:36.947 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-16T07:44:37.024 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-16T07:44:37.024 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-16T07:44:37.215 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-16T07:44:37.347 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T07:44:37.411 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/2 2024-04-16T07:44:38.079 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/2 2024-04-16T07:44:38.079 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/2 2024-04-16T07:44:38.385 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T07:44:38.385 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:38.385 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-16T07:44:38.385 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T07:44:38.385 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:38.385 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:44:38.623 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-volume 2024-04-16T07:44:39.221 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:07:00 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:44:39.378 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:44:39.379 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:44:39.379 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-04-16T07:44:39.379 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:44:39.379 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-16T07:44:39.379 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 273 k 2024-04-16T07:44:39.379 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-04-16T07:44:39.379 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-16T07:44:39.380 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:39.380 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-16T07:44:39.380 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:44:39.380 INFO:teuthology.orchestra.run.smithi070.stdout:Install 2 Packages 2024-04-16T07:44:39.380 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:39.380 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 342 k 2024-04-16T07:44:39.381 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 1.5 M 2024-04-16T07:44:39.381 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-16T07:44:39.502 INFO:teuthology.orchestra.run.smithi070.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 566 kB/s | 69 kB 00:00 2024-04-16T07:44:41.186 INFO:teuthology.orchestra.run.smithi070.stdout:(2/2): ceph-volume-19.0.0-3109.g30b0aa00.el9.no 151 kB/s | 273 kB 00:01 2024-04-16T07:44:41.187 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:44:41.187 INFO:teuthology.orchestra.run.smithi070.stdout:Total 189 kB/s | 342 kB 00:01 2024-04-16T07:44:41.188 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-16T07:44:41.198 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-16T07:44:41.198 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-16T07:44:41.239 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-16T07:44:41.239 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-16T07:44:41.455 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-16T07:44:41.656 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-16T07:44:41.696 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2/2 2024-04-16T07:44:41.712 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2/2 2024-04-16T07:44:41.713 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:44:41.713 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-16T07:44:41.713 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:42.485 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 1/2 2024-04-16T07:44:42.835 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-16T07:44:42.835 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:42.835 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-16T07:44:42.835 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:44:42.835 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-16T07:44:42.835 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:42.835 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:44:43.143 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install librados-devel 2024-04-16T07:44:43.741 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:07:04 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:44:43.892 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:44:43.893 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:44:43.893 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-04-16T07:44:43.893 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:44:43.894 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-16T07:44:43.894 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 125 k 2024-04-16T07:44:43.894 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:43.894 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-16T07:44:43.894 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:44:43.894 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-04-16T07:44:43.894 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:43.894 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 125 k 2024-04-16T07:44:43.894 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 445 k 2024-04-16T07:44:43.894 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-16T07:44:44.700 INFO:teuthology.orchestra.run.smithi070.stdout:librados-devel-19.0.0-3109.g30b0aa00.el9.x86_64 155 kB/s | 125 kB 00:00 2024-04-16T07:44:44.701 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:44:44.701 INFO:teuthology.orchestra.run.smithi070.stdout:Total 155 kB/s | 125 kB 00:00 2024-04-16T07:44:44.701 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-16T07:44:44.707 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-16T07:44:44.707 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-16T07:44:44.762 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-16T07:44:44.763 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-16T07:44:44.914 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-16T07:44:45.007 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:45.478 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:45.786 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:45.786 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:45.786 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-16T07:44:45.786 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:44:45.786 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:45.786 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:44:46.047 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install libcephfs2 2024-04-16T07:44:46.660 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:07:07 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:44:46.767 INFO:teuthology.orchestra.run.smithi070.stdout:Package libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:44:46.820 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:44:46.821 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-04-16T07:44:46.821 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:44:46.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install libcephfs-devel 2024-04-16T07:44:47.465 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:07:08 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:44:47.617 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:44:47.618 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:44:47.618 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-04-16T07:44:47.618 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:44:47.618 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-16T07:44:47.618 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 32 k 2024-04-16T07:44:47.618 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:47.618 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-16T07:44:47.618 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:44:47.618 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-04-16T07:44:47.619 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:47.619 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 32 k 2024-04-16T07:44:47.619 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 142 k 2024-04-16T07:44:47.619 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-16T07:44:48.062 INFO:teuthology.orchestra.run.smithi070.stdout:libcephfs-devel-19.0.0-3109.g30b0aa00.el9.x86_6 71 kB/s | 32 kB 00:00 2024-04-16T07:44:48.063 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:44:48.063 INFO:teuthology.orchestra.run.smithi070.stdout:Total 71 kB/s | 32 kB 00:00 2024-04-16T07:44:48.063 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-16T07:44:48.067 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-16T07:44:48.067 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-16T07:44:48.093 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-16T07:44:48.094 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-16T07:44:48.180 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-16T07:44:48.294 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:48.941 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:48.941 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:48.941 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-16T07:44:48.941 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:44:48.941 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:48.941 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:44:49.140 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install librados2 2024-04-16T07:44:49.720 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:07:10 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:44:49.820 INFO:teuthology.orchestra.run.smithi070.stdout:Package librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:44:49.872 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:44:49.873 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-04-16T07:44:49.873 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:44:49.942 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install librbd1 2024-04-16T07:44:50.524 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:07:11 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:44:50.623 INFO:teuthology.orchestra.run.smithi070.stdout:Package librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:44:50.674 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:44:50.676 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-04-16T07:44:50.676 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:44:50.738 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-rados 2024-04-16T07:44:51.317 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:07:12 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:44:51.417 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:44:51.469 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:44:51.469 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-04-16T07:44:51.470 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:44:51.533 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-rgw 2024-04-16T07:44:52.118 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:07:13 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:44:52.218 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:44:52.271 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:44:52.272 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-04-16T07:44:52.272 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:44:52.330 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-cephfs 2024-04-16T07:44:52.913 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:07:13 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:44:53.012 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:44:53.065 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:44:53.066 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-04-16T07:44:53.066 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:44:53.133 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-rbd 2024-04-16T07:44:53.717 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:07:14 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:44:53.817 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:44:53.869 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:44:53.870 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-04-16T07:44:53.870 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:44:53.949 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install rbd-fuse 2024-04-16T07:44:54.524 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:07:15 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:44:54.674 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:44:54.675 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:44:54.675 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2024-04-16T07:44:54.675 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:44:54.675 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-16T07:44:54.675 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 90 k 2024-04-16T07:44:54.675 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:54.675 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-16T07:44:54.675 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:44:54.675 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-04-16T07:44:54.676 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:54.676 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 90 k 2024-04-16T07:44:54.676 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 313 k 2024-04-16T07:44:54.676 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-16T07:44:55.390 INFO:teuthology.orchestra.run.smithi070.stdout:rbd-fuse-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 126 kB/s | 90 kB 00:00 2024-04-16T07:44:55.390 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:44:55.391 INFO:teuthology.orchestra.run.smithi070.stdout:Total 125 kB/s | 90 kB 00:00 2024-04-16T07:44:55.391 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-16T07:44:55.400 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-16T07:44:55.400 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-16T07:44:55.456 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-16T07:44:55.457 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-16T07:44:55.598 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-16T07:44:55.688 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:56.125 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:56.408 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:56.408 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:56.408 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-16T07:44:56.408 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:44:56.408 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:56.408 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:44:56.670 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install rbd-mirror 2024-04-16T07:44:57.261 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:07:18 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:44:57.416 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:44:57.417 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:44:57.417 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-04-16T07:44:57.417 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:44:57.417 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-16T07:44:57.417 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.4 M 2024-04-16T07:44:57.417 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:57.417 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-16T07:44:57.417 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:44:57.417 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-04-16T07:44:57.417 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:44:57.418 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 3.4 M 2024-04-16T07:44:57.418 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 29 M 2024-04-16T07:44:57.418 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-16T07:45:01.493 INFO:teuthology.orchestra.run.smithi070.stdout:rbd-mirror-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 855 kB/s | 3.4 MB 00:04 2024-04-16T07:45:01.494 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:45:01.494 INFO:teuthology.orchestra.run.smithi070.stdout:Total 854 kB/s | 3.4 MB 00:04 2024-04-16T07:45:01.495 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-16T07:45:01.507 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-16T07:45:01.507 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-16T07:45:01.585 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-16T07:45:01.586 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-16T07:45:02.148 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-16T07:45:02.218 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:45:02.243 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:45:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:45:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T07:45:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:45:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:45:02.243 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:45:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:03.556 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-16T07:45:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:45:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:03.556 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:45:03.816 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install rbd-nbd 2024-04-16T07:45:04.417 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:07:25 ago on Tue 16 Apr 2024 07:37:39 AM UTC. 2024-04-16T07:45:04.578 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-16T07:45:04.579 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:45:04.579 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2024-04-16T07:45:04.579 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:45:04.579 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-16T07:45:04.579 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 175 k 2024-04-16T07:45:04.580 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:04.580 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-16T07:45:04.580 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-16T07:45:04.580 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-04-16T07:45:04.580 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:04.580 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 175 k 2024-04-16T07:45:04.580 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 673 k 2024-04-16T07:45:04.580 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-16T07:45:05.422 INFO:teuthology.orchestra.run.smithi070.stdout:rbd-nbd-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 208 kB/s | 175 kB 00:00 2024-04-16T07:45:05.423 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:45:05.423 INFO:teuthology.orchestra.run.smithi070.stdout:Total 208 kB/s | 175 kB 00:00 2024-04-16T07:45:05.424 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-16T07:45:05.434 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-16T07:45:05.435 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-16T07:45:05.493 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-16T07:45:05.494 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-16T07:45:05.668 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-16T07:45:05.779 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:45:06.213 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:45:06.521 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:45:06.521 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:06.521 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-16T07:45:06.521 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:45:06.521 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:06.521 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-16T07:45:06.793 DEBUG:teuthology.parallel:result is None 2024-04-16T07:45:06.794 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:45:06.947 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T07:45:06.970 INFO:teuthology.orchestra.run.smithi070.stdout:19.0.0-3109.g30b0aa00.el9 2024-04-16T07:45:06.971 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3109.g30b0aa00.el9 2024-04-16T07:45:06.971 INFO:teuthology.task.install:The correct ceph version 19.0.0-3109.g30b0aa00 is installed. 2024-04-16T07:45:06.973 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:45:07.114 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T07:45:07.139 INFO:teuthology.orchestra.run.smithi172.stdout:19.0.0-3109.g30b0aa00.el9 2024-04-16T07:45:07.140 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3109.g30b0aa00.el9 2024-04-16T07:45:07.140 INFO:teuthology.task.install:The correct ceph version 19.0.0-3109.g30b0aa00 is installed. 2024-04-16T07:45:07.141 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-16T07:45:07.141 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-16T07:45:07.142 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T07:45:07.171 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-16T07:45:07.171 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T07:45:07.213 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-16T07:45:07.213 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-16T07:45:07.213 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T07:45:07.241 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T07:45:07.309 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-16T07:45:07.309 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T07:45:07.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T07:45:07.410 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-16T07:45:07.410 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-16T07:45:07.410 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T07:45:07.438 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T07:45:07.521 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-16T07:45:07.521 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T07:45:07.551 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T07:45:07.620 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-16T07:45:07.620 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-16T07:45:07.620 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T07:45:07.648 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T07:45:07.716 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-16T07:45:07.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T07:45:07.744 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T07:45:07.812 INFO:teuthology.run_tasks:Running task ceph... 2024-04-16T07:45:07.892 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-16T07:45:07.892 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 777 /var/log/ceph 2024-04-16T07:45:07.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /var/log/ceph 2024-04-16T07:45:07.923 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-16T07:45:07.923 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-16T07:45:07.961 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-16T07:45:07.991 INFO:tasks.ceph:Creating extra log directories... 2024-04-16T07:45:07.992 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-16T07:45:08.030 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-16T07:45:08.069 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-16T07:45:08.069 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-16T07:45:08.069 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657976', 'branch': 'wip-matanb-crimson-replicated-april-15', 'description': 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed}', 'email': None, 'first_in_suite': False, 'job_id': '7657976', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-replicated-april-15'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-matanb-crimson-replicated-april-15', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 7594, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-matanb-crimson-replicated-april-15', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'targets': {'smithi070.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMsJ2TJdaZN5y4l3u/X2+t7XwFtINmipTx2SWpFb/+DHE7lIYCT/PxtPsat5G3qtz/BI6Ro1PNqxidKlyqL4WcI=', 'smithi172.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMWPjIiC9oiD/C0llv5iLJ6LthJN1mBh2pA/Bpd0Ecmd+PM/Bw6gew0/xXSdD3v7vbwk6YRVobobOscRqkt5Ums='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'rados': {'clients': ['client.0'], 'objects': 500, 'op_weights': {'write': 100}, 'ops': 4000, 'write_fadvise_dontneed': True}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-16_07:22:04', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-16T07:45:08.070 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-16T07:45:08.099 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-16T07:45:08.115 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T07:45:08.154 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T07:45:08.184 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-16T07:45:08.184 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T07:45:08.235 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T07:45:08.235 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_1 2024-04-16T07:45:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T07:45:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:45:08.291 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-16T07:45:08.291 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:45:08.291 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:45:08.291 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-16 07:45:02.538506046 +0000 2024-04-16T07:45:08.291 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-16 07:36:51.556069914 +0000 2024-04-16T07:45:08.291 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-16 07:36:51.556069914 +0000 2024-04-16T07:45:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-16 07:36:51.556069914 +0000 2024-04-16T07:45:08.291 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T07:45:08.357 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-16T07:45:08.357 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-16T07:45:08.357 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 9.8573e-05 s, 5.2 MB/s 2024-04-16T07:45:08.358 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T07:45:08.414 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_2 2024-04-16T07:45:08.470 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T07:45:08.470 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:45:08.470 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-16T07:45:08.470 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:45:08.470 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:45:08.470 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-16 07:45:02.539506015 +0000 2024-04-16T07:45:08.470 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-16 07:36:51.895059527 +0000 2024-04-16T07:45:08.470 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-16 07:36:51.895059527 +0000 2024-04-16T07:45:08.470 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-16 07:36:51.895059527 +0000 2024-04-16T07:45:08.470 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T07:45:08.536 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-16T07:45:08.536 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-16T07:45:08.536 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 7.8294e-05 s, 6.5 MB/s 2024-04-16T07:45:08.537 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T07:45:08.593 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_3 2024-04-16T07:45:08.648 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T07:45:08.649 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:45:08.649 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-16T07:45:08.649 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:45:08.649 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:45:08.649 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-16 07:45:02.539506015 +0000 2024-04-16T07:45:08.649 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-16 07:36:52.238049019 +0000 2024-04-16T07:45:08.649 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-16 07:36:52.238049019 +0000 2024-04-16T07:45:08.649 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-16 07:36:52.238049019 +0000 2024-04-16T07:45:08.649 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T07:45:08.715 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-16T07:45:08.715 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-16T07:45:08.715 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 9.3957e-05 s, 5.4 MB/s 2024-04-16T07:45:08.716 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T07:45:08.772 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_4 2024-04-16T07:45:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T07:45:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:45:08.828 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-16T07:45:08.828 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:45:08.828 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:45:08.828 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-16 07:45:02.539506015 +0000 2024-04-16T07:45:08.828 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-16 07:36:52.582038481 +0000 2024-04-16T07:45:08.828 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-16 07:36:52.582038481 +0000 2024-04-16T07:45:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-16 07:36:52.582038481 +0000 2024-04-16T07:45:08.828 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T07:45:08.894 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-16T07:45:08.894 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-16T07:45:08.894 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000103816 s, 4.9 MB/s 2024-04-16T07:45:08.896 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T07:45:08.952 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-16T07:45:08.953 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-16T07:45:08.953 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T07:45:08.969 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T07:45:08.969 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2024-04-16T07:45:09.024 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T07:45:09.025 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:45:09.025 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-16T07:45:09.025 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:45:09.025 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:45:09.025 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-04-16 07:43:50.452254665 +0000 2024-04-16T07:45:09.025 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-04-16 07:36:52.024999597 +0000 2024-04-16T07:45:09.025 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-04-16 07:36:52.024999597 +0000 2024-04-16T07:45:09.025 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-04-16 07:36:52.024999597 +0000 2024-04-16T07:45:09.025 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T07:45:09.092 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-04-16T07:45:09.092 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-04-16T07:45:09.092 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000260236 s, 2.0 MB/s 2024-04-16T07:45:09.093 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T07:45:09.150 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2024-04-16T07:45:09.205 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T07:45:09.205 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:45:09.205 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-16T07:45:09.205 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:45:09.205 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:45:09.205 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-04-16 07:43:50.452254665 +0000 2024-04-16T07:45:09.205 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-04-16 07:36:52.377991331 +0000 2024-04-16T07:45:09.205 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-04-16 07:36:52.377991331 +0000 2024-04-16T07:45:09.205 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-04-16 07:36:52.377991331 +0000 2024-04-16T07:45:09.205 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T07:45:09.272 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-04-16T07:45:09.272 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-04-16T07:45:09.272 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.00021283 s, 2.4 MB/s 2024-04-16T07:45:09.273 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T07:45:09.329 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2024-04-16T07:45:09.385 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T07:45:09.385 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:45:09.385 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-16T07:45:09.385 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:45:09.385 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:45:09.385 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-04-16 07:43:50.453254642 +0000 2024-04-16T07:45:09.385 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-04-16 07:36:52.734982970 +0000 2024-04-16T07:45:09.386 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-04-16 07:36:52.734982970 +0000 2024-04-16T07:45:09.386 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-04-16 07:36:52.734982970 +0000 2024-04-16T07:45:09.386 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T07:45:09.452 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-04-16T07:45:09.452 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-04-16T07:45:09.452 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000221689 s, 2.3 MB/s 2024-04-16T07:45:09.453 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T07:45:09.509 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2024-04-16T07:45:09.564 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T07:45:09.564 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:45:09.565 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-16T07:45:09.565 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:45:09.565 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:45:09.565 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-04-16 07:43:50.453254642 +0000 2024-04-16T07:45:09.565 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-04-16 07:36:53.083974798 +0000 2024-04-16T07:45:09.565 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-04-16 07:36:53.083974798 +0000 2024-04-16T07:45:09.565 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-04-16 07:36:53.083974798 +0000 2024-04-16T07:45:09.565 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T07:45:09.630 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-04-16T07:45:09.631 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-04-16T07:45:09.631 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000188674 s, 2.7 MB/s 2024-04-16T07:45:09.632 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T07:45:09.688 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-16T07:45:09.688 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi070.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-04-16T07:45:09.689 INFO:tasks.ceph:Generating config... 2024-04-16T07:45:09.690 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-16T07:45:09.690 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-16T07:45:09.690 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-16T07:45:09.690 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-16T07:45:09.690 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-16T07:45:09.690 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-04-16T07:45:09.690 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-16T07:45:09.690 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-16T07:45:09.690 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-16T07:45:09.690 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-16T07:45:09.690 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-16T07:45:09.690 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-16T07:45:09.691 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-16T07:45:09.691 INFO:tasks.ceph:[osd] err to stderr = True 2024-04-16T07:45:09.691 INFO:tasks.ceph:[osd] log flush on exit = True 2024-04-16T07:45:09.691 INFO:tasks.ceph:[osd] log to file = False 2024-04-16T07:45:09.691 INFO:tasks.ceph:[osd] log to stderr = True 2024-04-16T07:45:09.691 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-16T07:45:09.691 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-16T07:45:09.691 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-16T07:45:09.691 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-16T07:45:09.691 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-04-16T07:45:09.691 INFO:tasks.ceph:[osd] osd delete sleep = 1 2024-04-16T07:45:09.691 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-16T07:45:09.691 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-04-16T07:45:09.691 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-16T07:45:09.691 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-16T07:45:09.691 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-04-16T07:45:09.692 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-04-16T07:45:09.692 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-16T07:45:09.692 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-04-16T07:45:09.692 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-16T07:45:09.692 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-16T07:45:09.692 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-16T07:45:09.692 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-16T07:45:09.692 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-16T07:45:09.692 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-16T07:45:09.692 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-16T07:45:09.692 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-04-16T07:45:09.692 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2024-04-16T07:45:09.692 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-16T07:45:09.692 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-16T07:45:09.692 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-16T07:45:09.692 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-16T07:45:09.692 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-16T07:45:09.693 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-04-16T07:45:09.693 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-16T07:45:09.693 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-16T07:45:09.693 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-16T07:45:09.693 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-16T07:45:09.693 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-04-16T07:45:09.693 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-04-16T07:45:09.693 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-04-16T07:45:09.693 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-04-16T07:45:09.693 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-04-16T07:45:09.693 INFO:tasks.ceph:Setting up mon.a... 2024-04-16T07:45:09.693 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-16T07:45:09.761 INFO:teuthology.orchestra.run.smithi070.stdout:creating /etc/ceph/ceph.keyring 2024-04-16T07:45:09.765 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-16T07:45:09.820 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T07:45:09.847 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.70')] 2024-04-16T07:45:09.848 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.70', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-04-16T07:45:09.848 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-16T07:45:09.849 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T07:45:09.902 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.70 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:45:09.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:09.992+0000 7f2d2ba37f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:09.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:09.992+0000 7f2d2ba37f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:09.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:09.993+0000 7f2d2ba37f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:09.993 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:45:09.993 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool: generated fsid e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6 2024-04-16T07:45:09.993 INFO:teuthology.orchestra.run.smithi070.stdout:setting min_mon_release = quincy 2024-04-16T07:45:09.993 INFO:teuthology.orchestra.run.smithi070.stdout:epoch 0 2024-04-16T07:45:09.993 INFO:teuthology.orchestra.run.smithi070.stdout:fsid e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6 2024-04-16T07:45:09.993 INFO:teuthology.orchestra.run.smithi070.stdout:last_changed 2024-04-16T07:45:09.994217+0000 2024-04-16T07:45:09.993 INFO:teuthology.orchestra.run.smithi070.stdout:created 2024-04-16T07:45:09.994217+0000 2024-04-16T07:45:09.993 INFO:teuthology.orchestra.run.smithi070.stdout:min_mon_release 17 (quincy) 2024-04-16T07:45:09.994 INFO:teuthology.orchestra.run.smithi070.stdout:election_strategy: 1 2024-04-16T07:45:09.994 INFO:teuthology.orchestra.run.smithi070.stdout:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-16T07:45:09.994 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-16T07:45:09.995 DEBUG:teuthology.orchestra.run.smithi070:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T07:45:10.009 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6... 2024-04-16T07:45:10.010 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-16T07:45:10.052 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-16T07:45:10.102 INFO:teuthology.orchestra.run.smithi070.stdout:[global] 2024-04-16T07:45:10.102 INFO:teuthology.orchestra.run.smithi070.stdout: chdir = "" 2024-04-16T07:45:10.102 INFO:teuthology.orchestra.run.smithi070.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T07:45:10.102 INFO:teuthology.orchestra.run.smithi070.stdout: auth supported = cephx 2024-04-16T07:45:10.103 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.103 INFO:teuthology.orchestra.run.smithi070.stdout: filestore xattr use omap = true 2024-04-16T07:45:10.103 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.103 INFO:teuthology.orchestra.run.smithi070.stdout: mon clock drift allowed = 1.000 2024-04-16T07:45:10.103 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.103 INFO:teuthology.orchestra.run.smithi070.stdout: osd crush chooseleaf type = 0 2024-04-16T07:45:10.103 INFO:teuthology.orchestra.run.smithi070.stdout: auth debug = true 2024-04-16T07:45:10.103 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.103 INFO:teuthology.orchestra.run.smithi070.stdout: ms die on old message = true 2024-04-16T07:45:10.103 INFO:teuthology.orchestra.run.smithi070.stdout: ms die on bug = true 2024-04-16T07:45:10.103 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.103 INFO:teuthology.orchestra.run.smithi070.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T07:45:10.103 INFO:teuthology.orchestra.run.smithi070.stdout: mon pg warn max object skew = 0 2024-04-16T07:45:10.103 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.103 INFO:teuthology.orchestra.run.smithi070.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T07:45:10.104 INFO:teuthology.orchestra.run.smithi070.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T07:45:10.104 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.104 INFO:teuthology.orchestra.run.smithi070.stdout: osd pool default size = 2 2024-04-16T07:45:10.104 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.104 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd allow primary affinity = true 2024-04-16T07:45:10.104 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd allow pg remap = true 2024-04-16T07:45:10.104 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on legacy crush tunables = false 2024-04-16T07:45:10.104 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on crush straw calc version zero = false 2024-04-16T07:45:10.104 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on no sortbitwise = false 2024-04-16T07:45:10.104 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on osd down out interval zero = false 2024-04-16T07:45:10.104 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on too few osds = false 2024-04-16T07:45:10.104 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T07:45:10.104 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T07:45:10.104 INFO:teuthology.orchestra.run.smithi070.stdout: mon_allow_pool_size_one = true 2024-04-16T07:45:10.104 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.104 INFO:teuthology.orchestra.run.smithi070.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T07:45:10.105 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.105 INFO:teuthology.orchestra.run.smithi070.stdout: osd default data pool replay window = 5 2024-04-16T07:45:10.105 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.105 INFO:teuthology.orchestra.run.smithi070.stdout: mon allow pool delete = true 2024-04-16T07:45:10.105 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.105 INFO:teuthology.orchestra.run.smithi070.stdout: mon cluster log file level = debug 2024-04-16T07:45:10.105 INFO:teuthology.orchestra.run.smithi070.stdout: debug asserts on shutdown = true 2024-04-16T07:45:10.105 INFO:teuthology.orchestra.run.smithi070.stdout: mon health detail to clog = false 2024-04-16T07:45:10.105 INFO:teuthology.orchestra.run.smithi070.stdout: mon host = 172.21.15.70 2024-04-16T07:45:10.105 INFO:teuthology.orchestra.run.smithi070.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-16T07:45:10.105 INFO:teuthology.orchestra.run.smithi070.stdout: ms client mode = crc 2024-04-16T07:45:10.105 INFO:teuthology.orchestra.run.smithi070.stdout: ms cluster mode = crc 2024-04-16T07:45:10.105 INFO:teuthology.orchestra.run.smithi070.stdout: ms mon client mode = crc 2024-04-16T07:45:10.105 INFO:teuthology.orchestra.run.smithi070.stdout: ms mon cluster mode = crc 2024-04-16T07:45:10.105 INFO:teuthology.orchestra.run.smithi070.stdout: ms mon service mode = crc 2024-04-16T07:45:10.106 INFO:teuthology.orchestra.run.smithi070.stdout: ms service mode = crc 2024-04-16T07:45:10.106 INFO:teuthology.orchestra.run.smithi070.stdout: osd_pool_default_min_size = 2 2024-04-16T07:45:10.106 INFO:teuthology.orchestra.run.smithi070.stdout: osd_pool_default_size = 3 2024-04-16T07:45:10.106 INFO:teuthology.orchestra.run.smithi070.stdout: fsid = e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6 2024-04-16T07:45:10.106 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.106 INFO:teuthology.orchestra.run.smithi070.stdout:[osd] 2024-04-16T07:45:10.106 INFO:teuthology.orchestra.run.smithi070.stdout: osd journal size = 100 2024-04-16T07:45:10.106 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.106 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub load threshold = 5.0 2024-04-16T07:45:10.106 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub max interval = 120 2024-04-16T07:45:10.106 INFO:teuthology.orchestra.run.smithi070.stdout: osd mclock profile = high_recovery_ops 2024-04-16T07:45:10.106 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.106 INFO:teuthology.orchestra.run.smithi070.stdout: osd recover clone overlap = true 2024-04-16T07:45:10.106 INFO:teuthology.orchestra.run.smithi070.stdout: osd recovery max chunk = 1048576 2024-04-16T07:45:10.107 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.107 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug shutdown = true 2024-04-16T07:45:10.107 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug op order = true 2024-04-16T07:45:10.107 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug verify stray on activate = true 2024-04-16T07:45:10.107 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug trim objects = true 2024-04-16T07:45:10.107 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.107 INFO:teuthology.orchestra.run.smithi070.stdout: osd open classes on start = true 2024-04-16T07:45:10.107 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug pg log writeout = true 2024-04-16T07:45:10.107 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.107 INFO:teuthology.orchestra.run.smithi070.stdout: osd deep scrub update digest min age = 30 2024-04-16T07:45:10.107 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.107 INFO:teuthology.orchestra.run.smithi070.stdout: osd map max advance = 10 2024-04-16T07:45:10.107 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.107 INFO:teuthology.orchestra.run.smithi070.stdout: journal zero on create = true 2024-04-16T07:45:10.107 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.107 INFO:teuthology.orchestra.run.smithi070.stdout: filestore ondisk finisher threads = 3 2024-04-16T07:45:10.107 INFO:teuthology.orchestra.run.smithi070.stdout: filestore apply finisher threads = 3 2024-04-16T07:45:10.108 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.108 INFO:teuthology.orchestra.run.smithi070.stdout: bdev debug aio = true 2024-04-16T07:45:10.108 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug misdirected ops = true 2024-04-16T07:45:10.108 INFO:teuthology.orchestra.run.smithi070.stdout: debug monc = 20 2024-04-16T07:45:10.108 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore block size = 96636764160 2024-04-16T07:45:10.108 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore compression algorithm = snappy 2024-04-16T07:45:10.108 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore compression mode = aggressive 2024-04-16T07:45:10.108 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore fsck on mount = True 2024-04-16T07:45:10.108 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore rocksdb cf = False 2024-04-16T07:45:10.108 INFO:teuthology.orchestra.run.smithi070.stdout: crimson alien thread cpu cores = 6-7 2024-04-16T07:45:10.108 INFO:teuthology.orchestra.run.smithi070.stdout: crimson osd obc lru size = 10 2024-04-16T07:45:10.108 INFO:teuthology.orchestra.run.smithi070.stdout: debug bluefs = 20 2024-04-16T07:45:10.108 INFO:teuthology.orchestra.run.smithi070.stdout: debug bluestore = 20 2024-04-16T07:45:10.108 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-04-16T07:45:10.108 INFO:teuthology.orchestra.run.smithi070.stdout: debug osd = 20 2024-04-16T07:45:10.109 INFO:teuthology.orchestra.run.smithi070.stdout: debug rocksdb = 10 2024-04-16T07:45:10.109 INFO:teuthology.orchestra.run.smithi070.stdout: err to stderr = True 2024-04-16T07:45:10.109 INFO:teuthology.orchestra.run.smithi070.stdout: log flush on exit = True 2024-04-16T07:45:10.109 INFO:teuthology.orchestra.run.smithi070.stdout: log to file = False 2024-04-16T07:45:10.109 INFO:teuthology.orchestra.run.smithi070.stdout: log to stderr = True 2024-04-16T07:45:10.109 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd backfillfull_ratio = 0.85 2024-04-16T07:45:10.109 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd full ratio = 0.9 2024-04-16T07:45:10.109 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd nearfull ratio = 0.8 2024-04-16T07:45:10.109 INFO:teuthology.orchestra.run.smithi070.stdout: osd blocked scrub grace period = 3600 2024-04-16T07:45:10.109 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug reject backfill probability = 0.3 2024-04-16T07:45:10.109 INFO:teuthology.orchestra.run.smithi070.stdout: osd delete sleep = 1 2024-04-16T07:45:10.109 INFO:teuthology.orchestra.run.smithi070.stdout: osd failsafe full ratio = 0.95 2024-04-16T07:45:10.109 INFO:teuthology.orchestra.run.smithi070.stdout: osd max backfills = 3 2024-04-16T07:45:10.109 INFO:teuthology.orchestra.run.smithi070.stdout: osd max markdown count = 1000 2024-04-16T07:45:10.109 INFO:teuthology.orchestra.run.smithi070.stdout: osd objectstore = bluestore 2024-04-16T07:45:10.109 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub min interval = 60 2024-04-16T07:45:10.110 INFO:teuthology.orchestra.run.smithi070.stdout: osd shutdown pgref assert = True 2024-04-16T07:45:10.110 INFO:teuthology.orchestra.run.smithi070.stdout: osd snap trim sleep = 2 2024-04-16T07:45:10.110 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.110 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr] 2024-04-16T07:45:10.110 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-04-16T07:45:10.110 INFO:teuthology.orchestra.run.smithi070.stdout: debug mgr = 20 2024-04-16T07:45:10.110 INFO:teuthology.orchestra.run.smithi070.stdout: debug mon = 20 2024-04-16T07:45:10.110 INFO:teuthology.orchestra.run.smithi070.stdout: debug auth = 20 2024-04-16T07:45:10.110 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min pgs per osd = 4 2024-04-16T07:45:10.110 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min bytes per osd = 10 2024-04-16T07:45:10.110 INFO:teuthology.orchestra.run.smithi070.stdout: mgr/telemetry/nag = false 2024-04-16T07:45:10.110 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.110 INFO:teuthology.orchestra.run.smithi070.stdout:[mon] 2024-04-16T07:45:10.110 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-04-16T07:45:10.110 INFO:teuthology.orchestra.run.smithi070.stdout: debug mon = 20 2024-04-16T07:45:10.111 INFO:teuthology.orchestra.run.smithi070.stdout: debug paxos = 20 2024-04-16T07:45:10.111 INFO:teuthology.orchestra.run.smithi070.stdout: debug auth = 20 2024-04-16T07:45:10.111 INFO:teuthology.orchestra.run.smithi070.stdout: mon data avail warn = 5 2024-04-16T07:45:10.111 INFO:teuthology.orchestra.run.smithi070.stdout: mon mgr mkfs grace = 240 2024-04-16T07:45:10.111 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min pgs per osd = 4 2024-04-16T07:45:10.114 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd reporter subtree level = osd 2024-04-16T07:45:10.114 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd prime pg temp = true 2024-04-16T07:45:10.115 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min bytes per osd = 10 2024-04-16T07:45:10.115 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.115 INFO:teuthology.orchestra.run.smithi070.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T07:45:10.115 INFO:teuthology.orchestra.run.smithi070.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T07:45:10.115 INFO:teuthology.orchestra.run.smithi070.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T07:45:10.115 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.115 INFO:teuthology.orchestra.run.smithi070.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T07:45:10.115 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T07:45:10.115 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T07:45:10.115 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.115 INFO:teuthology.orchestra.run.smithi070.stdout: # 1m isn't quite enough 2024-04-16T07:45:10.115 INFO:teuthology.orchestra.run.smithi070.stdout: mon_down_mkfs_grace = 2m 2024-04-16T07:45:10.115 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.115 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_filestore_osds = false 2024-04-16T07:45:10.115 INFO:teuthology.orchestra.run.smithi070.stdout: mon min osdmap epochs = 50 2024-04-16T07:45:10.115 INFO:teuthology.orchestra.run.smithi070.stdout: mon osdmap full prune interval = 2 2024-04-16T07:45:10.115 INFO:teuthology.orchestra.run.smithi070.stdout: mon osdmap full prune min = 15 2024-04-16T07:45:10.116 INFO:teuthology.orchestra.run.smithi070.stdout: mon osdmap full prune txsize = 2 2024-04-16T07:45:10.116 INFO:teuthology.orchestra.run.smithi070.stdout: osd pool default crimson = True 2024-04-16T07:45:10.116 INFO:teuthology.orchestra.run.smithi070.stdout: paxos service trim min = 10 2024-04-16T07:45:10.116 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:10.116 INFO:teuthology.orchestra.run.smithi070.stdout:[client] 2024-04-16T07:45:10.116 INFO:teuthology.orchestra.run.smithi070.stdout: rgw cache enabled = true 2024-04-16T07:45:10.116 INFO:teuthology.orchestra.run.smithi070.stdout: rgw enable ops log = true 2024-04-16T07:45:10.116 INFO:teuthology.orchestra.run.smithi070.stdout: rgw enable usage log = true 2024-04-16T07:45:10.116 INFO:teuthology.orchestra.run.smithi070.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T07:45:10.116 INFO:teuthology.orchestra.run.smithi070.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T07:45:10.116 INFO:teuthology.orchestra.run.smithi070.stdout:[mon.a] 2024-04-16T07:45:10.116 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.0] 2024-04-16T07:45:10.116 INFO:teuthology.orchestra.run.smithi070.stdout: crimson seastar cpu cores = 0-2 2024-04-16T07:45:10.116 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.1] 2024-04-16T07:45:10.116 INFO:teuthology.orchestra.run.smithi070.stdout: crimson seastar cpu cores = 3-5 2024-04-16T07:45:10.116 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.2] 2024-04-16T07:45:10.117 INFO:teuthology.orchestra.run.smithi070.stdout: crimson seastar cpu cores = 0-2 2024-04-16T07:45:10.117 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.3] 2024-04-16T07:45:10.117 INFO:teuthology.orchestra.run.smithi070.stdout: crimson seastar cpu cores = 3-5 2024-04-16T07:45:10.117 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2024-04-16T07:45:10.117 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2024-04-16T07:45:10.117 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T07:45:10.117 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2024-04-16T07:45:10.117 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.118 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2024-04-16T07:45:10.118 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.118 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2024-04-16T07:45:10.118 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.118 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2024-04-16T07:45:10.118 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2024-04-16T07:45:10.118 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.118 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2024-04-16T07:45:10.118 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2024-04-16T07:45:10.118 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.118 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T07:45:10.118 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2024-04-16T07:45:10.118 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.118 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T07:45:10.118 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T07:45:10.119 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.119 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2024-04-16T07:45:10.119 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.119 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2024-04-16T07:45:10.119 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2024-04-16T07:45:10.119 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2024-04-16T07:45:10.119 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2024-04-16T07:45:10.119 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2024-04-16T07:45:10.119 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2024-04-16T07:45:10.119 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2024-04-16T07:45:10.119 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T07:45:10.119 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T07:45:10.119 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2024-04-16T07:45:10.119 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.119 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T07:45:10.120 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.120 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2024-04-16T07:45:10.120 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.120 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2024-04-16T07:45:10.120 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.120 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2024-04-16T07:45:10.120 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2024-04-16T07:45:10.120 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2024-04-16T07:45:10.120 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = 172.21.15.70 2024-04-16T07:45:10.120 INFO:teuthology.orchestra.run.smithi172.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-16T07:45:10.120 INFO:teuthology.orchestra.run.smithi172.stdout: ms client mode = crc 2024-04-16T07:45:10.120 INFO:teuthology.orchestra.run.smithi172.stdout: ms cluster mode = crc 2024-04-16T07:45:10.120 INFO:teuthology.orchestra.run.smithi172.stdout: ms mon client mode = crc 2024-04-16T07:45:10.120 INFO:teuthology.orchestra.run.smithi172.stdout: ms mon cluster mode = crc 2024-04-16T07:45:10.120 INFO:teuthology.orchestra.run.smithi172.stdout: ms mon service mode = crc 2024-04-16T07:45:10.121 INFO:teuthology.orchestra.run.smithi172.stdout: ms service mode = crc 2024-04-16T07:45:10.121 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_min_size = 2 2024-04-16T07:45:10.121 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_size = 3 2024-04-16T07:45:10.121 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6 2024-04-16T07:45:10.121 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.121 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2024-04-16T07:45:10.121 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2024-04-16T07:45:10.121 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.121 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2024-04-16T07:45:10.121 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 120 2024-04-16T07:45:10.121 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock profile = high_recovery_ops 2024-04-16T07:45:10.121 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.121 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2024-04-16T07:45:10.121 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2024-04-16T07:45:10.121 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.121 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2024-04-16T07:45:10.122 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2024-04-16T07:45:10.122 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2024-04-16T07:45:10.122 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug trim objects = true 2024-04-16T07:45:10.122 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.122 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2024-04-16T07:45:10.122 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2024-04-16T07:45:10.122 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.122 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2024-04-16T07:45:10.122 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.122 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2024-04-16T07:45:10.122 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.122 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2024-04-16T07:45:10.122 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.122 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2024-04-16T07:45:10.122 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2024-04-16T07:45:10.123 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.123 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2024-04-16T07:45:10.123 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2024-04-16T07:45:10.123 INFO:teuthology.orchestra.run.smithi172.stdout: debug monc = 20 2024-04-16T07:45:10.123 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore block size = 96636764160 2024-04-16T07:45:10.123 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore compression algorithm = snappy 2024-04-16T07:45:10.123 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore compression mode = aggressive 2024-04-16T07:45:10.123 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore fsck on mount = True 2024-04-16T07:45:10.123 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore rocksdb cf = False 2024-04-16T07:45:10.123 INFO:teuthology.orchestra.run.smithi172.stdout: crimson alien thread cpu cores = 6-7 2024-04-16T07:45:10.123 INFO:teuthology.orchestra.run.smithi172.stdout: crimson osd obc lru size = 10 2024-04-16T07:45:10.123 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluefs = 20 2024-04-16T07:45:10.123 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluestore = 20 2024-04-16T07:45:10.123 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-04-16T07:45:10.123 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 20 2024-04-16T07:45:10.124 INFO:teuthology.orchestra.run.smithi172.stdout: debug rocksdb = 10 2024-04-16T07:45:10.124 INFO:teuthology.orchestra.run.smithi172.stdout: err to stderr = True 2024-04-16T07:45:10.124 INFO:teuthology.orchestra.run.smithi172.stdout: log flush on exit = True 2024-04-16T07:45:10.124 INFO:teuthology.orchestra.run.smithi172.stdout: log to file = False 2024-04-16T07:45:10.124 INFO:teuthology.orchestra.run.smithi172.stdout: log to stderr = True 2024-04-16T07:45:10.124 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd backfillfull_ratio = 0.85 2024-04-16T07:45:10.124 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd full ratio = 0.9 2024-04-16T07:45:10.124 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd nearfull ratio = 0.8 2024-04-16T07:45:10.124 INFO:teuthology.orchestra.run.smithi172.stdout: osd blocked scrub grace period = 3600 2024-04-16T07:45:10.124 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug reject backfill probability = 0.3 2024-04-16T07:45:10.124 INFO:teuthology.orchestra.run.smithi172.stdout: osd delete sleep = 1 2024-04-16T07:45:10.124 INFO:teuthology.orchestra.run.smithi172.stdout: osd failsafe full ratio = 0.95 2024-04-16T07:45:10.124 INFO:teuthology.orchestra.run.smithi172.stdout: osd max backfills = 3 2024-04-16T07:45:10.124 INFO:teuthology.orchestra.run.smithi172.stdout: osd max markdown count = 1000 2024-04-16T07:45:10.124 INFO:teuthology.orchestra.run.smithi172.stdout: osd objectstore = bluestore 2024-04-16T07:45:10.124 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub min interval = 60 2024-04-16T07:45:10.125 INFO:teuthology.orchestra.run.smithi172.stdout: osd shutdown pgref assert = True 2024-04-16T07:45:10.125 INFO:teuthology.orchestra.run.smithi172.stdout: osd snap trim sleep = 2 2024-04-16T07:45:10.125 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.125 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2024-04-16T07:45:10.125 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-04-16T07:45:10.125 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2024-04-16T07:45:10.125 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-04-16T07:45:10.125 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-04-16T07:45:10.125 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-04-16T07:45:10.125 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-04-16T07:45:10.125 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2024-04-16T07:45:10.125 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.125 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2024-04-16T07:45:10.125 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-04-16T07:45:10.125 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-04-16T07:45:10.126 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2024-04-16T07:45:10.126 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-04-16T07:45:10.126 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2024-04-16T07:45:10.126 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2024-04-16T07:45:10.126 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-04-16T07:45:10.126 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2024-04-16T07:45:10.126 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2024-04-16T07:45:10.126 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-04-16T07:45:10.126 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.126 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T07:45:10.126 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T07:45:10.126 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T07:45:10.126 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.126 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T07:45:10.126 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T07:45:10.127 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T07:45:10.127 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.127 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2024-04-16T07:45:10.127 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2024-04-16T07:45:10.127 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.127 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_filestore_osds = false 2024-04-16T07:45:10.127 INFO:teuthology.orchestra.run.smithi172.stdout: mon min osdmap epochs = 50 2024-04-16T07:45:10.127 INFO:teuthology.orchestra.run.smithi172.stdout: mon osdmap full prune interval = 2 2024-04-16T07:45:10.127 INFO:teuthology.orchestra.run.smithi172.stdout: mon osdmap full prune min = 15 2024-04-16T07:45:10.127 INFO:teuthology.orchestra.run.smithi172.stdout: mon osdmap full prune txsize = 2 2024-04-16T07:45:10.127 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default crimson = True 2024-04-16T07:45:10.127 INFO:teuthology.orchestra.run.smithi172.stdout: paxos service trim min = 10 2024-04-16T07:45:10.127 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-16T07:45:10.127 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2024-04-16T07:45:10.127 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2024-04-16T07:45:10.128 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2024-04-16T07:45:10.128 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2024-04-16T07:45:10.128 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T07:45:10.128 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T07:45:10.128 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2024-04-16T07:45:10.128 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2024-04-16T07:45:10.128 INFO:teuthology.orchestra.run.smithi172.stdout: crimson seastar cpu cores = 0-2 2024-04-16T07:45:10.128 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2024-04-16T07:45:10.128 INFO:teuthology.orchestra.run.smithi172.stdout: crimson seastar cpu cores = 3-5 2024-04-16T07:45:10.128 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2024-04-16T07:45:10.128 INFO:teuthology.orchestra.run.smithi172.stdout: crimson seastar cpu cores = 0-2 2024-04-16T07:45:10.128 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.3] 2024-04-16T07:45:10.128 INFO:teuthology.orchestra.run.smithi172.stdout: crimson seastar cpu cores = 3-5 2024-04-16T07:45:10.130 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-16T07:45:10.130 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-16T07:45:10.196 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-16T07:45:10.196 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-16T07:45:10.196 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-16T07:45:10.211 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-16T07:45:10.211 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-16T07:45:10.266 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi070.front.sepia.ceph.com 2024-04-16T07:45:10.266 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-16T07:45:10.266 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-16T07:45:10.267 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T07:45:10.347 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-16T07:45:10.347 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:45:10.402 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2024-04-16T07:45:10.403 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-16T07:45:10.403 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-16T07:45:10.403 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T07:45:10.444 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-16T07:45:10.444 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:45:10.500 INFO:tasks.ceph:Setting up mon nodes... 2024-04-16T07:45:10.500 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-16T07:45:10.500 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-16T07:45:10.609 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-16T07:45:10.612 INFO:tasks.ceph:Setting up mds nodes... 2024-04-16T07:45:10.612 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-16T07:45:10.612 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-16T07:45:10.667 INFO:teuthology.orchestra.run.smithi070.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-16T07:45:10.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-16T07:45:10.736 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-16T07:45:10.753 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-16T07:45:10.753 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi070.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-04-16T07:45:10.753 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-16T07:45:10.781 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-16T07:45:10.781 INFO:tasks.ceph:role: osd.0 2024-04-16T07:45:10.781 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-16T07:45:10.781 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-16T07:45:10.848 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T07:45:10.849 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T07:45:10.849 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T07:45:10.849 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T07:45:10.849 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T07:45:10.849 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-04-16T07:45:10.849 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T07:45:10.849 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T07:45:10.849 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T07:45:10.849 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T07:45:10.881 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-04-16T07:45:10.884 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-04-16T07:45:10.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-16T07:45:10.956 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-16T07:45:11.032 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-16T07:45:11.101 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-16T07:45:11.101 INFO:tasks.ceph:role: osd.1 2024-04-16T07:45:11.101 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-16T07:45:11.101 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-16T07:45:11.169 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T07:45:11.169 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T07:45:11.169 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T07:45:11.169 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T07:45:11.169 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T07:45:11.169 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-04-16T07:45:11.169 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T07:45:11.170 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T07:45:11.170 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T07:45:11.170 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T07:45:11.202 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-04-16T07:45:11.204 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-04-16T07:45:11.204 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-16T07:45:11.275 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-16T07:45:11.347 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:45:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:45:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-04-16 07:45:11,661 seastar - Reactor backend: linux-aio 2024-04-16T07:45:11.670 INFO:teuthology.orchestra.run.smithi070.stderr:==29120==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:45:11.795 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-04-16 07:45:11,796 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T07:45:11.795 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-04-16 07:45:11,796 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-04-16T07:45:11.795 INFO:teuthology.orchestra.run.smithi070.stderr:DEBUG 2024-04-16 07:45:11,797 [shard 0:main] seastar - reactor::drain 2024-04-16T07:45:12.332 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-04-16T07:45:12.332 INFO:teuthology.orchestra.run.smithi070.stderr:Suppressions used: 2024-04-16T07:45:12.332 INFO:teuthology.orchestra.run.smithi070.stderr: count bytes template 2024-04-16T07:45:12.332 INFO:teuthology.orchestra.run.smithi070.stderr: 1 8 MallocExtension::Register 2024-04-16T07:45:12.332 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-04-16T07:45:12.332 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-16T07:45:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:45:12.402 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-04-16 07:45:12,403 seastar - Reactor backend: linux-aio 2024-04-16T07:45:12.411 INFO:teuthology.orchestra.run.smithi070.stderr:==29118==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:45:12.508 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb99d759 0xb870220 0xb87179c 0xb8719b2 0xb871b08 0xb871fd1 0x54daf 0x113cb0 0x4a0bd 0x35b7c 0x3629e 0x5dbe6f4 0x5dbe8f0 0x5dbeaed 0x5dbedc3 0x5e0d603 0x5e0d809 0x5d8c2f0 0x5d8c824 0x5bd1519 0x5bec5dd 0x6ec6e84 0x6ed8657 0x6ed9af4 0x6ed9b79 0x6ed9c7b 0x6ed9d73 0x6ed9f80 0x6eda166 0x6eda4b5 0x6eda859 0x6edaa89 0x6edacf3 0x6edafc0 0x6edb1de 0x6edb296 0x6edb35c 0x6edb43d 0x5a39e18 0x5a39f22 0x5a39fde 0x5a3a09a 0x5a3a156 0x5a3a2a3 0x5a3a42e 0x5a3a7ef 0x5a56730 0xb882ea5 0xb89d13f 0xb93e13d 0xb93f78b 0xb61bf71 0xb61c8ed 0x367a090 0x3feaf 0x3ff5f 0x3463444 2024-04-16T07:45:12.508 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-04-16T07:45:17.051 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-04-16T07:45:17.051 INFO:teuthology.orchestra.run.smithi070.stderr:Suppressions used: 2024-04-16T07:45:17.051 INFO:teuthology.orchestra.run.smithi070.stderr: count bytes template 2024-04-16T07:45:17.051 INFO:teuthology.orchestra.run.smithi070.stderr: 1 8 MallocExtension::Register 2024-04-16T07:45:17.051 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-04-16T07:45:17.051 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-16T07:45:17.052 INFO:teuthology.orchestra.run.smithi070.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6 2024-04-16T07:45:17.075 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T07:45:17.104 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:45:17.400 INFO:teuthology.orchestra.run.smithi070.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:45:17.401 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-04-16 07:45:17,403 seastar - Reactor backend: linux-aio 2024-04-16T07:45:17.406 INFO:teuthology.orchestra.run.smithi070.stderr:==29302==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:45:17.530 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-04-16 07:45:17,531 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T07:45:17.530 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-04-16 07:45:17,531 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-04-16T07:45:17.530 INFO:teuthology.orchestra.run.smithi070.stderr:DEBUG 2024-04-16 07:45:17,532 [shard 0:main] seastar - reactor::drain 2024-04-16T07:45:18.057 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-04-16T07:45:18.057 INFO:teuthology.orchestra.run.smithi070.stderr:Suppressions used: 2024-04-16T07:45:18.057 INFO:teuthology.orchestra.run.smithi070.stderr: count bytes template 2024-04-16T07:45:18.057 INFO:teuthology.orchestra.run.smithi070.stderr: 1 8 MallocExtension::Register 2024-04-16T07:45:18.057 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-04-16T07:45:18.057 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-16T07:45:18.113 INFO:teuthology.orchestra.run.smithi070.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:45:18.117 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-04-16 07:45:18,118 seastar - Reactor backend: linux-aio 2024-04-16T07:45:18.131 INFO:teuthology.orchestra.run.smithi070.stderr:==29300==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:45:22.772 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-04-16T07:45:22.772 INFO:teuthology.orchestra.run.smithi070.stderr:Suppressions used: 2024-04-16T07:45:22.772 INFO:teuthology.orchestra.run.smithi070.stderr: count bytes template 2024-04-16T07:45:22.773 INFO:teuthology.orchestra.run.smithi070.stderr: 1 8 MallocExtension::Register 2024-04-16T07:45:22.773 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-04-16T07:45:22.773 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-16T07:45:22.773 INFO:teuthology.orchestra.run.smithi070.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6 2024-04-16T07:45:22.795 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T07:45:22.824 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-16T07:45:22.858 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-16T07:45:22.858 INFO:tasks.ceph:role: osd.2 2024-04-16T07:45:22.858 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com 2024-04-16T07:45:22.858 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-16T07:45:22.938 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T07:45:22.938 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T07:45:22.938 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T07:45:22.938 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T07:45:22.939 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T07:45:22.939 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-04-16T07:45:22.939 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T07:45:22.939 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T07:45:22.939 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T07:45:22.939 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T07:45:23.457 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-04-16T07:45:23.460 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-04-16T07:45:23.460 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-04-16T07:45:23.617 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-16T07:45:23.657 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-16T07:45:23.729 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-16T07:45:23.729 INFO:tasks.ceph:role: osd.3 2024-04-16T07:45:23.730 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com 2024-04-16T07:45:23.730 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-16T07:45:23.807 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T07:45:23.807 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T07:45:23.807 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T07:45:23.807 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T07:45:23.807 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T07:45:23.808 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-04-16T07:45:23.808 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T07:45:23.808 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T07:45:23.808 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T07:45:23.808 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T07:45:24.259 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-04-16T07:45:24.261 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-04-16T07:45:24.262 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-04-16T07:45:24.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-16T07:45:24.384 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:45:24.720 INFO:teuthology.orchestra.run.smithi172.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:45:24.721 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-04-16 07:45:24,723 seastar - Reactor backend: linux-aio 2024-04-16T07:45:24.738 INFO:teuthology.orchestra.run.smithi172.stderr:==28990==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:45:24.869 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-04-16 07:45:24,870 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T07:45:24.869 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-04-16 07:45:24,870 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-04-16T07:45:24.870 INFO:teuthology.orchestra.run.smithi172.stderr:DEBUG 2024-04-16 07:45:24,871 [shard 0:main] seastar - reactor::drain 2024-04-16T07:45:25.438 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-16T07:45:25.438 INFO:teuthology.orchestra.run.smithi172.stderr:Suppressions used: 2024-04-16T07:45:25.438 INFO:teuthology.orchestra.run.smithi172.stderr: count bytes template 2024-04-16T07:45:25.438 INFO:teuthology.orchestra.run.smithi172.stderr: 1 8 MallocExtension::Register 2024-04-16T07:45:25.438 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-16T07:45:25.438 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-04-16T07:45:25.497 INFO:teuthology.orchestra.run.smithi172.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:45:25.501 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-04-16 07:45:25,502 seastar - Reactor backend: linux-aio 2024-04-16T07:45:25.510 INFO:teuthology.orchestra.run.smithi172.stderr:==28988==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:45:25.575 INFO:teuthology.orchestra.run.smithi172.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb99d759 0xb870220 0xb87179c 0xb8719b2 0xb871b08 0xb871fd1 0x54daf 0xc6026 0x5c2330d 0x5bec4be 0x6ec6e84 0x6ed8657 0x6ed9af4 0x6ed9b79 0x6ed9c7b 0x6ed9d73 0x6ed9f80 0x6eda166 0x6eda4b5 0x6eda859 0x6edaa89 0x6edacf3 0x6edafc0 0x6edb1de 0x6edb296 0x6edb35c 0x6edb43d 0x5a39e18 0x5a39f22 0x5a39fde 0x5a3a09a 0x5a3a156 0x5a3a2a3 0x5a3a42e 0x5a3a7ef 0x5a56730 0xb882ea5 0xb89d13f 0xb93e13d 0xb93f78b 0xb61bf71 0xb61c8ed 0x367a090 0x3feaf 0x3ff5f 0x3463444 2024-04-16T07:45:25.575 INFO:teuthology.orchestra.run.smithi172.stderr:kernel callstack: 2024-04-16T07:45:30.181 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-16T07:45:30.181 INFO:teuthology.orchestra.run.smithi172.stderr:Suppressions used: 2024-04-16T07:45:30.181 INFO:teuthology.orchestra.run.smithi172.stderr: count bytes template 2024-04-16T07:45:30.181 INFO:teuthology.orchestra.run.smithi172.stderr: 1 8 MallocExtension::Register 2024-04-16T07:45:30.181 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-16T07:45:30.181 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-04-16T07:45:30.181 INFO:teuthology.orchestra.run.smithi172.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6 2024-04-16T07:45:30.203 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T07:45:30.232 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:45:30.545 INFO:teuthology.orchestra.run.smithi172.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:45:30.546 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-04-16 07:45:30,548 seastar - Reactor backend: linux-aio 2024-04-16T07:45:30.551 INFO:teuthology.orchestra.run.smithi172.stderr:==29173==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:45:30.676 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-04-16 07:45:30,678 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T07:45:30.677 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-04-16 07:45:30,678 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-04-16T07:45:30.677 INFO:teuthology.orchestra.run.smithi172.stderr:DEBUG 2024-04-16 07:45:30,679 [shard 0:main] seastar - reactor::drain 2024-04-16T07:45:31.222 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-16T07:45:31.222 INFO:teuthology.orchestra.run.smithi172.stderr:Suppressions used: 2024-04-16T07:45:31.222 INFO:teuthology.orchestra.run.smithi172.stderr: count bytes template 2024-04-16T07:45:31.222 INFO:teuthology.orchestra.run.smithi172.stderr: 1 8 MallocExtension::Register 2024-04-16T07:45:31.222 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-16T07:45:31.222 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-04-16T07:45:31.288 INFO:teuthology.orchestra.run.smithi172.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:45:31.292 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-04-16 07:45:31,293 seastar - Reactor backend: linux-aio 2024-04-16T07:45:31.301 INFO:teuthology.orchestra.run.smithi172.stderr:==29171==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:45:35.963 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-16T07:45:35.963 INFO:teuthology.orchestra.run.smithi172.stderr:Suppressions used: 2024-04-16T07:45:35.963 INFO:teuthology.orchestra.run.smithi172.stderr: count bytes template 2024-04-16T07:45:35.963 INFO:teuthology.orchestra.run.smithi172.stderr: 1 8 MallocExtension::Register 2024-04-16T07:45:35.963 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-16T07:45:35.963 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-04-16T07:45:35.964 INFO:teuthology.orchestra.run.smithi172.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6 2024-04-16T07:45:35.985 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T07:45:36.014 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-16T07:45:36.014 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-16T07:45:36.015 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-16T07:45:36.048 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-16T07:45:36.048 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-16T07:45:36.120 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-16T07:45:36.120 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-16T07:45:36.149 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-16T07:45:36.149 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-16T07:45:36.217 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-16T07:45:36.217 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-16T07:45:36.286 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-16T07:45:36.286 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-16T07:45:36.304 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-16T07:45:36.304 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-16T07:45:36.344 INFO:tasks.ceph:Adding keys to all mons... 2024-04-16T07:45:36.344 DEBUG:teuthology.orchestra.run.smithi070:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-16T07:45:36.374 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.0] 2024-04-16T07:45:36.375 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQCILB5mQAD9IRAAynzLUcFN+OmIeFx6hqf2iw== 2024-04-16T07:45:36.375 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.1] 2024-04-16T07:45:36.375 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQCOLB5meZqfEBAArRYMdhzWC6dlthqvfNM+lA== 2024-04-16T07:45:36.375 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr.x] 2024-04-16T07:45:36.375 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQCGLB5m7elsJBAA2oJfgR8dE3R42IlRAe/uOA== 2024-04-16T07:45:36.375 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.2] 2024-04-16T07:45:36.375 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQCVLB5m1ESbJxAAEmQeGB3Ala9xcMX+poYMEg== 2024-04-16T07:45:36.375 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.3] 2024-04-16T07:45:36.375 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQCbLB5mAy8FGxAAwQoL/Py0Ui2tiK9/Ye0hdA== 2024-04-16T07:45:36.375 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2024-04-16T07:45:36.375 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQCGLB5mri/eJxAAP3OOJyQdXA8PkeK2RbCMMg== 2024-04-16T07:45:36.375 INFO:teuthology.orchestra.run.smithi070.stdout:[client.1] 2024-04-16T07:45:36.375 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQCGLB5m9CMALBAASwnVDA164xheTbqvwZSEFw== 2024-04-16T07:45:36.376 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T07:45:36.482 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T07:45:36.545 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-16T07:45:36.609 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T07:45:36.673 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T07:45:36.739 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T07:45:36.809 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T07:45:36.875 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-16T07:45:36.876 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-16T07:45:36.903 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-16T07:45:37.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:37.016+0000 7f2c7e630a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:37.016+0000 7f2c7e630a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.029 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-16T07:45:37.057 DEBUG:teuthology.orchestra.run.smithi070:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:45:37.112 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-16T07:45:37.112 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-16T07:45:37.113 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-16T07:45:37.154 INFO:tasks.ceph.mon.a:Started 2024-04-16T07:45:37.155 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-16T07:45:37.155 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-16T07:45:37.155 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-16T07:45:37.158 INFO:tasks.ceph.mgr.x:Started 2024-04-16T07:45:37.158 INFO:tasks.ceph:Setting crush tunables to default 2024-04-16T07:45:37.158 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd crush tunables default 2024-04-16T07:45:37.241 INFO:tasks.ceph.mon.a.smithi070.stderr:2024-04-16T07:45:37.241+0000 7f3fd7e6ea80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.241 INFO:tasks.ceph.mon.a.smithi070.stderr:2024-04-16T07:45:37.241+0000 7f3fd7e6ea80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.241 INFO:tasks.ceph.mon.a.smithi070.stderr:2024-04-16T07:45:37.242+0000 7f3fd7e6ea80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.263 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:37.263+0000 7f4e8df77000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.277 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:37.277+0000 7f4e8df77000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.277 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:37.277+0000 7f4e8df77000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:37.271+0000 7f5fe919d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:37.281+0000 7f5fe919d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.353 INFO:teuthology.orchestra.run.smithi070.stderr:adjusted tunables profile to default 2024-04-16T07:45:37.365 INFO:tasks.ceph:check_enable_crimson: True 2024-04-16T07:45:37.365 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-16T07:45:37.365 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-16T07:45:37.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:37.471+0000 7f814619d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:37.480+0000 7f814619d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.561 INFO:tasks.ceph.mon.a.smithi070.stderr:2024-04-16T07:45:37.562+0000 7f3fcebf4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-16T07:45:37.561 INFO:tasks.ceph.mon.a.smithi070.stderr:Please be aware that this feature is experimental, untested, 2024-04-16T07:45:37.561 INFO:tasks.ceph.mon.a.smithi070.stderr:unsupported, and may result in data corruption, data loss, 2024-04-16T07:45:37.561 INFO:tasks.ceph.mon.a.smithi070.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-16T07:45:37.561 INFO:tasks.ceph.mon.a.smithi070.stderr:feature with important data. 2024-04-16T07:45:37.562 INFO:tasks.ceph.mon.a.smithi070.stderr: 2024-04-16T07:45:37.745 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:37.745+0000 7f4e8df77000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T07:45:37.799 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:37.799+0000 7f4e8df77000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T07:45:38.322 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:38.322+0000 7f4e8df77000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T07:45:38.368 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-16T07:45:38.368 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-16T07:45:38.369 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-16T07:45:38.375 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:38.376+0000 7f4e8df77000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T07:45:38.399 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-16T07:45:38.399 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-16T07:45:38.419 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:38.420+0000 7f4e8df77000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T07:45:38.469 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-16T07:45:38.469 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-16T07:45:38.473 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:38.474+0000 7f4e8df77000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T07:45:38.498 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-16T07:45:38.498 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-16T07:45:38.518 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:38.519+0000 7f4e8df77000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T07:45:38.546 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 816d6f55-bf1d-48a9-9552-fc285b77358e 0 2024-04-16T07:45:38.642 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:38.643+0000 7f4e8df77000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T07:45:38.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-16T07:45:38.694+0000 7fad0199d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:38.702 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-16T07:45:38.703+0000 7fad0199d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:38.717 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:38.718+0000 7f4e8df77000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T07:45:38.765 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:38.766+0000 7f4e8df77000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T07:45:38.782 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-04-16T07:45:38.794 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new f777184a-794a-4817-ba31-872e243adcd4 1 2024-04-16T07:45:38.821 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:38.822+0000 7f4e8df77000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T07:45:38.872 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:38.873+0000 7f4e8df77000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T07:45:38.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-16T07:45:38.901+0000 7f060ea6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:38.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-16T07:45:38.910+0000 7f060ea6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:38.989 INFO:teuthology.orchestra.run.smithi172.stdout:1 2024-04-16T07:45:39.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 340d1b05-a919-45a7-bc75-1cb38943c8e2 2 2024-04-16T07:45:39.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-16T07:45:39.148+0000 7f1cdf8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:39.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-16T07:45:39.157+0000 7f1cdf8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:39.236 INFO:teuthology.orchestra.run.smithi172.stdout:2 2024-04-16T07:45:39.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 659133e9-2b60-43c8-9034-f4690c837fe9 3 2024-04-16T07:45:39.274 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:39.274+0000 7f4e8df77000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T07:45:39.361 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-16T07:45:39.356+0000 7f817cd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:39.364 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-16T07:45:39.365+0000 7f817cd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:39.409 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:39.410+0000 7f4e8df77000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T07:45:39.445 INFO:teuthology.orchestra.run.smithi172.stdout:3 2024-04-16T07:45:39.456 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-16T07:45:39.495 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-16T07:45:39.495 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-16T07:45:39.537 INFO:tasks.ceph.osd.0:Started 2024-04-16T07:45:39.537 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-16T07:45:39.540 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:39.541+0000 7f4e8df77000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T07:45:39.574 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-16T07:45:39.574 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-16T07:45:39.586 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:39.586+0000 7f4e8df77000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T07:45:39.620 INFO:tasks.ceph.osd.1:Started 2024-04-16T07:45:39.620 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-16T07:45:39.653 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-16T07:45:39.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-16T07:45:39.695 INFO:tasks.ceph.osd.2:Started 2024-04-16T07:45:39.696 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-04-16T07:45:39.702 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:39.703+0000 7f4e8df77000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T07:45:39.728 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-16T07:45:39.728 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-16T07:45:39.771 INFO:tasks.ceph.osd.3:Started 2024-04-16T07:45:39.771 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:45:39.807 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:39.807+0000 7f4e8df77000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T07:45:39.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:39.888+0000 7fc264a64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:39.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:39.896+0000 7fc264a64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:39.964 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:39.964 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":7,"fsid":"e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6","created":"2024-04-16T07:45:37.265201+0000","modified":"2024-04-16T07:45:39.445562+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T07:45:39.445562+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"816d6f55-bf1d-48a9-9552-fc285b77358e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f777184a-794a-4817-ba31-872e243adcd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"340d1b05-a919-45a7-bc75-1cb38943c8e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"659133e9-2b60-43c8-9034-f4690c837fe9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:45:39.976 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-16T07:45:39.976 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-16T07:45:39.982 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:39.983+0000 7f4e8df77000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T07:45:40.029 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:40.030+0000 7f4e8df77000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T07:45:40.195 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:40.196+0000 7f4e8df77000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T07:45:40.379 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T07:45:40.481 INFO:teuthology.misc.health.smithi070.stderr:2024-04-16T07:45:40.478+0000 7fa91246d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:40.485 INFO:teuthology.misc.health.smithi070.stderr:2024-04-16T07:45:40.486+0000 7fa91246d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:40.555 INFO:teuthology.misc.health.smithi070.stdout: 2024-04-16T07:45:40.555 INFO:teuthology.misc.health.smithi070.stdout:{"epoch":7,"fsid":"e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6","created":"2024-04-16T07:45:37.265201+0000","modified":"2024-04-16T07:45:39.445562+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T07:45:39.445562+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"816d6f55-bf1d-48a9-9552-fc285b77358e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f777184a-794a-4817-ba31-872e243adcd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"340d1b05-a919-45a7-bc75-1cb38943c8e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"659133e9-2b60-43c8-9034-f4690c837fe9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:45:40.565 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-04-16T07:45:41.083 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:41.083+0000 7f4e8df77000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T07:45:41.370 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:41.370+0000 7f4e8df77000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T07:45:41.425 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:41.425+0000 7f4e8df77000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T07:45:41.610 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:41.611+0000 7f4e8df77000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T07:45:42.620 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-16T07:45:42.620+0000 7f4e2d07d640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-16T07:45:46.969 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T07:45:47.084 INFO:teuthology.misc.health.smithi070.stderr:2024-04-16T07:45:47.081+0000 7f1f6bc17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:47.088 INFO:teuthology.misc.health.smithi070.stderr:2024-04-16T07:45:47.089+0000 7f1f6bc17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:47.279 INFO:teuthology.misc.health.smithi070.stdout: 2024-04-16T07:45:47.279 INFO:teuthology.misc.health.smithi070.stdout:{"epoch":10,"fsid":"e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6","created":"2024-04-16T07:45:37.265201+0000","modified":"2024-04-16T07:45:46.626705+0000","last_up_change":"2024-04-16T07:45:44.621725+0000","last_in_change":"2024-04-16T07:45:39.445562+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:45:45.641324+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"816d6f55-bf1d-48a9-9552-fc285b77358e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6801","nonce":1984645931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1984645931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":1984645931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6811","nonce":1984645931}]},"public_addr":"172.21.15.70:6801/1984645931","cluster_addr":"172.21.15.70:6802/1984645931","heartbeat_back_addr":"172.21.15.70:6810/1984645931","heartbeat_front_addr":"172.21.15.70:6811/1984645931","state":["exists","up"]},{"osd":1,"uuid":"f777184a-794a-4817-ba31-872e243adcd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":873227738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6805","nonce":873227738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":873227738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6807","nonce":873227738}]},"public_addr":"172.21.15.70:6804/873227738","cluster_addr":"172.21.15.70:6805/873227738","heartbeat_back_addr":"172.21.15.70:6806/873227738","heartbeat_front_addr":"172.21.15.70:6807/873227738","state":["exists","up"]},{"osd":2,"uuid":"340d1b05-a919-45a7-bc75-1cb38943c8e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":911620790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":911620790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":911620790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":911620790}]},"public_addr":"172.21.15.172:6804/911620790","cluster_addr":"172.21.15.172:6805/911620790","heartbeat_back_addr":"172.21.15.172:6810/911620790","heartbeat_front_addr":"172.21.15.172:6809/911620790","state":["exists","up"]},{"osd":3,"uuid":"659133e9-2b60-43c8-9034-f4690c837fe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1749870805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6807","nonce":1749870805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":1749870805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6803","nonce":1749870805}]},"public_addr":"172.21.15.172:6808/1749870805","cluster_addr":"172.21.15.172:6807/1749870805","heartbeat_back_addr":"172.21.15.172:6806/1749870805","heartbeat_front_addr":"172.21.15.172:6803/1749870805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:45:47.291 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-04-16T07:45:47.291 INFO:tasks.ceph:Creating RBD pool 2024-04-16T07:45:47.291 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-16T07:45:47.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:47.396+0000 7f6bc02ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:47.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:47.407+0000 7f6bc02ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:47.629 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'rbd' created 2024-04-16T07:45:47.655 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-16T07:45:47.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:47.776+0000 7f738959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:47.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:47.789+0000 7f738959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:48.636 INFO:teuthology.orchestra.run.smithi070.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-16T07:45:48.657 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-16T07:45:48.657 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-16T07:45:48.657 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-16T07:45:48.658 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-16T07:45:48.658 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-16T07:45:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:48.800+0000 7f8ce075d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:48.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:48.810+0000 7f8ce075d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:49.075 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:49.088 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":4,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":842102803},{"type":"v1","addr":"172.21.15.172:6801","nonce":842102803}]},"active_addr":"172.21.15.172:6801/842102803","active_change":"2024-04-16T07:45:41.614321+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":2857280629}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":693849985}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":2531292704}]}]} 2024-04-16T07:45:49.092 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-16T07:45:49.092 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-16T07:45:49.092 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:45:49.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:49.210+0000 7f4fe40ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:49.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:49.219+0000 7f4fe40ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:49.425 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:49.425 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":12,"fsid":"e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6","created":"2024-04-16T07:45:37.265201+0000","modified":"2024-04-16T07:45:48.634278+0000","last_up_change":"2024-04-16T07:45:44.621725+0000","last_in_change":"2024-04-16T07:45:39.445562+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:45:45.641324+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T07:45:47.599173+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"816d6f55-bf1d-48a9-9552-fc285b77358e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6801","nonce":1984645931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1984645931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":1984645931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6811","nonce":1984645931}]},"public_addr":"172.21.15.70:6801/1984645931","cluster_addr":"172.21.15.70:6802/1984645931","heartbeat_back_addr":"172.21.15.70:6810/1984645931","heartbeat_front_addr":"172.21.15.70:6811/1984645931","state":["exists","up"]},{"osd":1,"uuid":"f777184a-794a-4817-ba31-872e243adcd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":873227738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6805","nonce":873227738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":873227738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6807","nonce":873227738}]},"public_addr":"172.21.15.70:6804/873227738","cluster_addr":"172.21.15.70:6805/873227738","heartbeat_back_addr":"172.21.15.70:6806/873227738","heartbeat_front_addr":"172.21.15.70:6807/873227738","state":["exists","up"]},{"osd":2,"uuid":"340d1b05-a919-45a7-bc75-1cb38943c8e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":911620790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":911620790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":911620790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":911620790}]},"public_addr":"172.21.15.172:6804/911620790","cluster_addr":"172.21.15.172:6805/911620790","heartbeat_back_addr":"172.21.15.172:6810/911620790","heartbeat_front_addr":"172.21.15.172:6809/911620790","state":["exists","up"]},{"osd":3,"uuid":"659133e9-2b60-43c8-9034-f4690c837fe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1749870805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6807","nonce":1749870805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":1749870805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6803","nonce":1749870805}]},"public_addr":"172.21.15.172:6808/1749870805","cluster_addr":"172.21.15.172:6807/1749870805","heartbeat_back_addr":"172.21.15.172:6806/1749870805","heartbeat_front_addr":"172.21.15.172:6803/1749870805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:45:49.437 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-16T07:45:49.438 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:45:49.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:49.546+0000 7f76b6ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:49.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:49.553+0000 7f76b6ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:49.748 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:49.748 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":13,"fsid":"e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6","created":"2024-04-16T07:45:37.265201+0000","modified":"2024-04-16T07:45:49.637253+0000","last_up_change":"2024-04-16T07:45:44.621725+0000","last_in_change":"2024-04-16T07:45:39.445562+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:45:45.641324+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T07:45:47.599173+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"816d6f55-bf1d-48a9-9552-fc285b77358e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6801","nonce":1984645931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1984645931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":1984645931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6811","nonce":1984645931}]},"public_addr":"172.21.15.70:6801/1984645931","cluster_addr":"172.21.15.70:6802/1984645931","heartbeat_back_addr":"172.21.15.70:6810/1984645931","heartbeat_front_addr":"172.21.15.70:6811/1984645931","state":["exists","up"]},{"osd":1,"uuid":"f777184a-794a-4817-ba31-872e243adcd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":873227738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6805","nonce":873227738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":873227738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6807","nonce":873227738}]},"public_addr":"172.21.15.70:6804/873227738","cluster_addr":"172.21.15.70:6805/873227738","heartbeat_back_addr":"172.21.15.70:6806/873227738","heartbeat_front_addr":"172.21.15.70:6807/873227738","state":["exists","up"]},{"osd":2,"uuid":"340d1b05-a919-45a7-bc75-1cb38943c8e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":911620790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":911620790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":911620790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":911620790}]},"public_addr":"172.21.15.172:6804/911620790","cluster_addr":"172.21.15.172:6805/911620790","heartbeat_back_addr":"172.21.15.172:6810/911620790","heartbeat_front_addr":"172.21.15.172:6809/911620790","state":["exists","up"]},{"osd":3,"uuid":"659133e9-2b60-43c8-9034-f4690c837fe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1749870805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6807","nonce":1749870805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":1749870805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6803","nonce":1749870805}]},"public_addr":"172.21.15.172:6808/1749870805","cluster_addr":"172.21.15.172:6807/1749870805","heartbeat_back_addr":"172.21.15.172:6806/1749870805","heartbeat_front_addr":"172.21.15.172:6803/1749870805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:45:49.763 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-16T07:45:49.763 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-16T07:45:49.763 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-16T07:45:49.764 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-16T07:45:49.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:49.881+0000 7fa078e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:49.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:49.881+0000 7f0391c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:49.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:49.881+0000 7f797899d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:49.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:49.881+0000 7f3e23aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:49.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:49.891+0000 7f3e23aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:49.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:49.892+0000 7f0391c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:49.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:49.892+0000 7fa078e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:49.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:49.893+0000 7f797899d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:49.927 INFO:teuthology.orchestra.run.smithi070.stdout:38654705669 2024-04-16T07:45:49.927 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T07:45:49.928 INFO:teuthology.orchestra.run.smithi070.stdout:38654705669 2024-04-16T07:45:49.928 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T07:45:49.929 INFO:teuthology.orchestra.run.smithi070.stdout:38654705669 2024-04-16T07:45:49.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-16T07:45:49.930 INFO:teuthology.orchestra.run.smithi070.stdout:38654705669 2024-04-16T07:45:49.930 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T07:45:50.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:50.047+0000 7f594a99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:50.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:50.047+0000 7f8b33a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:50.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:50.047+0000 7fa4ec5fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:50.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:50.047+0000 7f91e5c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:50.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:50.057+0000 7f91e5c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:50.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:50.057+0000 7f594a99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:50.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:50.057+0000 7fa4ec5fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:50.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:50.057+0000 7f8b33a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:50.265 INFO:teuthology.orchestra.run.smithi070.stdout:38654705669 2024-04-16T07:45:50.267 INFO:teuthology.orchestra.run.smithi070.stdout:38654705669 2024-04-16T07:45:50.271 INFO:teuthology.orchestra.run.smithi070.stdout:38654705669 2024-04-16T07:45:50.280 INFO:teuthology.orchestra.run.smithi070.stdout:38654705669 2024-04-16T07:45:50.281 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.0 2024-04-16T07:45:50.281 DEBUG:teuthology.parallel:result is None 2024-04-16T07:45:50.286 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.3 2024-04-16T07:45:50.287 DEBUG:teuthology.parallel:result is None 2024-04-16T07:45:50.287 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.1 2024-04-16T07:45:50.287 DEBUG:teuthology.parallel:result is None 2024-04-16T07:45:50.294 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.2 2024-04-16T07:45:50.294 DEBUG:teuthology.parallel:result is None 2024-04-16T07:45:50.294 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T07:45:50.294 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:45:50.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:50.444+0000 7fa436b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:50.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:50.451+0000 7fa436b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:50.634 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:50.635 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-16T07:45:50.648 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-16T07:45:49.653878+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107912,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379448,"statfs":{"total":386547056640,"available":386436554752,"internally_reserved":0,"allocated":593920,"data_stored":1870024,"data_compressed":89727,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.769721+0000","last_change":"2024-04-16T07:45:48.769721+0000","last_active":"2024-04-16T07:45:48.769721+0000","last_peered":"2024-04-16T07:45:48.769721+0000","last_clean":"2024-04-16T07:45:48.769721+0000","last_became_active":"2024-04-16T07:45:48.764515+0000","last_became_peered":"2024-04-16T07:45:48.764515+0000","last_unstale":"2024-04-16T07:45:48.769721+0000","last_undegraded":"2024-04-16T07:45:48.769721+0000","last_fullsized":"2024-04-16T07:45:48.769721+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.765070+0000","last_change":"2024-04-16T07:45:48.766237+0000","last_active":"2024-04-16T07:45:48.765070+0000","last_peered":"2024-04-16T07:45:48.765070+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.761479+0000","last_became_peered":"2024-04-16T07:45:48.761479+0000","last_unstale":"2024-04-16T07:45:48.765070+0000","last_undegraded":"2024-04-16T07:45:48.765070+0000","last_fullsized":"2024-04-16T07:45:48.765070+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.765103+0000","last_change":"2024-04-16T07:45:48.766846+0000","last_active":"2024-04-16T07:45:48.765103+0000","last_peered":"2024-04-16T07:45:48.765103+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.761593+0000","last_became_peered":"2024-04-16T07:45:48.761593+0000","last_unstale":"2024-04-16T07:45:48.765103+0000","last_undegraded":"2024-04-16T07:45:48.765103+0000","last_fullsized":"2024-04-16T07:45:48.765103+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.717426+0000","last_change":"2024-04-16T07:45:48.718558+0000","last_active":"2024-04-16T07:45:48.717426+0000","last_peered":"2024-04-16T07:45:48.717426+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.713866+0000","last_became_peered":"2024-04-16T07:45:48.713866+0000","last_unstale":"2024-04-16T07:45:48.717426+0000","last_undegraded":"2024-04-16T07:45:48.717426+0000","last_fullsized":"2024-04-16T07:45:48.717426+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.773933+0000","last_change":"2024-04-16T07:45:48.774974+0000","last_active":"2024-04-16T07:45:48.773933+0000","last_peered":"2024-04-16T07:45:48.773933+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.768713+0000","last_became_peered":"2024-04-16T07:45:48.768713+0000","last_unstale":"2024-04-16T07:45:48.773933+0000","last_undegraded":"2024-04-16T07:45:48.773933+0000","last_fullsized":"2024-04-16T07:45:48.773933+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:49.648519+0000","last_change":"2024-04-16T07:45:48.724954+0000","last_active":"2024-04-16T07:45:49.648519+0000","last_peered":"2024-04-16T07:45:49.648519+0000","last_clean":"2024-04-16T07:45:49.648519+0000","last_became_active":"2024-04-16T07:45:48.719985+0000","last_became_peered":"2024-04-16T07:45:48.719985+0000","last_unstale":"2024-04-16T07:45:49.648519+0000","last_undegraded":"2024-04-16T07:45:49.648519+0000","last_fullsized":"2024-04-16T07:45:49.648519+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.811632+0000","last_change":"2024-04-16T07:45:48.812462+0000","last_active":"2024-04-16T07:45:48.811632+0000","last_peered":"2024-04-16T07:45:48.811632+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.804250+0000","last_became_peered":"2024-04-16T07:45:48.804250+0000","last_unstale":"2024-04-16T07:45:48.811632+0000","last_undegraded":"2024-04-16T07:45:48.811632+0000","last_fullsized":"2024-04-16T07:45:48.811632+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.796564+0000","last_change":"2024-04-16T07:45:48.796564+0000","last_active":"2024-04-16T07:45:48.796564+0000","last_peered":"2024-04-16T07:45:48.796564+0000","last_clean":"2024-04-16T07:45:48.796564+0000","last_became_active":"2024-04-16T07:45:48.789023+0000","last_became_peered":"2024-04-16T07:45:48.789023+0000","last_unstale":"2024-04-16T07:45:48.796564+0000","last_undegraded":"2024-04-16T07:45:48.796564+0000","last_fullsized":"2024-04-16T07:45:48.796564+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:49.351146+0000","last_change":"2024-04-16T07:45:47.694188+0000","last_active":"2024-04-16T07:45:49.351146+0000","last_peered":"2024-04-16T07:45:49.351146+0000","last_clean":"2024-04-16T07:45:49.351146+0000","last_became_active":"2024-04-16T07:45:47.687698+0000","last_became_peered":"2024-04-16T07:45:47.687698+0000","last_unstale":"2024-04-16T07:45:49.351146+0000","last_undegraded":"2024-04-16T07:45:49.351146+0000","last_fullsized":"2024-04-16T07:45:49.351146+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_clean_scrub_stamp":"2024-04-16T07:45:46.626705+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:45:50.649 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:45:50.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:50.759+0000 7f76b126d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:50.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:50.766+0000 7f76b126d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:50.955 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:50.955 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-16T07:45:50.967 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-16T07:45:49.653878+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107912,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379448,"statfs":{"total":386547056640,"available":386436554752,"internally_reserved":0,"allocated":593920,"data_stored":1870024,"data_compressed":89727,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.769721+0000","last_change":"2024-04-16T07:45:48.769721+0000","last_active":"2024-04-16T07:45:48.769721+0000","last_peered":"2024-04-16T07:45:48.769721+0000","last_clean":"2024-04-16T07:45:48.769721+0000","last_became_active":"2024-04-16T07:45:48.764515+0000","last_became_peered":"2024-04-16T07:45:48.764515+0000","last_unstale":"2024-04-16T07:45:48.769721+0000","last_undegraded":"2024-04-16T07:45:48.769721+0000","last_fullsized":"2024-04-16T07:45:48.769721+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.765070+0000","last_change":"2024-04-16T07:45:48.766237+0000","last_active":"2024-04-16T07:45:48.765070+0000","last_peered":"2024-04-16T07:45:48.765070+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.761479+0000","last_became_peered":"2024-04-16T07:45:48.761479+0000","last_unstale":"2024-04-16T07:45:48.765070+0000","last_undegraded":"2024-04-16T07:45:48.765070+0000","last_fullsized":"2024-04-16T07:45:48.765070+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.765103+0000","last_change":"2024-04-16T07:45:48.766846+0000","last_active":"2024-04-16T07:45:48.765103+0000","last_peered":"2024-04-16T07:45:48.765103+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.761593+0000","last_became_peered":"2024-04-16T07:45:48.761593+0000","last_unstale":"2024-04-16T07:45:48.765103+0000","last_undegraded":"2024-04-16T07:45:48.765103+0000","last_fullsized":"2024-04-16T07:45:48.765103+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.717426+0000","last_change":"2024-04-16T07:45:48.718558+0000","last_active":"2024-04-16T07:45:48.717426+0000","last_peered":"2024-04-16T07:45:48.717426+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.713866+0000","last_became_peered":"2024-04-16T07:45:48.713866+0000","last_unstale":"2024-04-16T07:45:48.717426+0000","last_undegraded":"2024-04-16T07:45:48.717426+0000","last_fullsized":"2024-04-16T07:45:48.717426+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.773933+0000","last_change":"2024-04-16T07:45:48.774974+0000","last_active":"2024-04-16T07:45:48.773933+0000","last_peered":"2024-04-16T07:45:48.773933+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.768713+0000","last_became_peered":"2024-04-16T07:45:48.768713+0000","last_unstale":"2024-04-16T07:45:48.773933+0000","last_undegraded":"2024-04-16T07:45:48.773933+0000","last_fullsized":"2024-04-16T07:45:48.773933+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:49.648519+0000","last_change":"2024-04-16T07:45:48.724954+0000","last_active":"2024-04-16T07:45:49.648519+0000","last_peered":"2024-04-16T07:45:49.648519+0000","last_clean":"2024-04-16T07:45:49.648519+0000","last_became_active":"2024-04-16T07:45:48.719985+0000","last_became_peered":"2024-04-16T07:45:48.719985+0000","last_unstale":"2024-04-16T07:45:49.648519+0000","last_undegraded":"2024-04-16T07:45:49.648519+0000","last_fullsized":"2024-04-16T07:45:49.648519+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.811632+0000","last_change":"2024-04-16T07:45:48.812462+0000","last_active":"2024-04-16T07:45:48.811632+0000","last_peered":"2024-04-16T07:45:48.811632+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.804250+0000","last_became_peered":"2024-04-16T07:45:48.804250+0000","last_unstale":"2024-04-16T07:45:48.811632+0000","last_undegraded":"2024-04-16T07:45:48.811632+0000","last_fullsized":"2024-04-16T07:45:48.811632+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.796564+0000","last_change":"2024-04-16T07:45:48.796564+0000","last_active":"2024-04-16T07:45:48.796564+0000","last_peered":"2024-04-16T07:45:48.796564+0000","last_clean":"2024-04-16T07:45:48.796564+0000","last_became_active":"2024-04-16T07:45:48.789023+0000","last_became_peered":"2024-04-16T07:45:48.789023+0000","last_unstale":"2024-04-16T07:45:48.796564+0000","last_undegraded":"2024-04-16T07:45:48.796564+0000","last_fullsized":"2024-04-16T07:45:48.796564+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:49.351146+0000","last_change":"2024-04-16T07:45:47.694188+0000","last_active":"2024-04-16T07:45:49.351146+0000","last_peered":"2024-04-16T07:45:49.351146+0000","last_clean":"2024-04-16T07:45:49.351146+0000","last_became_active":"2024-04-16T07:45:47.687698+0000","last_became_peered":"2024-04-16T07:45:47.687698+0000","last_unstale":"2024-04-16T07:45:49.351146+0000","last_undegraded":"2024-04-16T07:45:49.351146+0000","last_fullsized":"2024-04-16T07:45:49.351146+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_clean_scrub_stamp":"2024-04-16T07:45:46.626705+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:45:50.968 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-16T07:45:50.968 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-16T07:45:50.968 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-16T07:45:50.969 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-16T07:45:51.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:51.078+0000 7f4797c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:51.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:51.084+0000 7f4797c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:51.316 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:51.316 INFO:teuthology.orchestra.run.smithi070.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-16T07:45:51.328 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-16T07:45:51.329 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-16T07:45:51.347 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-16T07:45:51.347 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-matanb-crimson-replicated-april-15'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-matanb-crimson-replicated-april-15', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}} 2024-04-16T07:45:51.347 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-16T07:45:51.347 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-16T07:45:51.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:51.456+0000 7f893aaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:51.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:51.467+0000 7f893aaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:51.667 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-16T07:45:51.667 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T07:45:51.667 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:45:51.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:51.776+0000 7fa3cde6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:51.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:51.783+0000 7fa3cde6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:51.969 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:51.969 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-16T07:45:51.982 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-16T07:45:51.619113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107912,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379448,"statfs":{"total":386547056640,"available":386436554752,"internally_reserved":0,"allocated":593920,"data_stored":1870024,"data_compressed":89727,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.965235"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:49.652256+0000","last_change":"2024-04-16T07:45:48.769721+0000","last_active":"2024-04-16T07:45:49.652256+0000","last_peered":"2024-04-16T07:45:49.652256+0000","last_clean":"2024-04-16T07:45:49.652256+0000","last_became_active":"2024-04-16T07:45:48.764515+0000","last_became_peered":"2024-04-16T07:45:48.764515+0000","last_unstale":"2024-04-16T07:45:49.652256+0000","last_undegraded":"2024-04-16T07:45:49.652256+0000","last_fullsized":"2024-04-16T07:45:49.652256+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.765070+0000","last_change":"2024-04-16T07:45:48.766237+0000","last_active":"2024-04-16T07:45:48.765070+0000","last_peered":"2024-04-16T07:45:48.765070+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.761479+0000","last_became_peered":"2024-04-16T07:45:48.761479+0000","last_unstale":"2024-04-16T07:45:48.765070+0000","last_undegraded":"2024-04-16T07:45:48.765070+0000","last_fullsized":"2024-04-16T07:45:48.765070+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.765103+0000","last_change":"2024-04-16T07:45:48.766846+0000","last_active":"2024-04-16T07:45:48.765103+0000","last_peered":"2024-04-16T07:45:48.765103+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.761593+0000","last_became_peered":"2024-04-16T07:45:48.761593+0000","last_unstale":"2024-04-16T07:45:48.765103+0000","last_undegraded":"2024-04-16T07:45:48.765103+0000","last_fullsized":"2024-04-16T07:45:48.765103+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.717426+0000","last_change":"2024-04-16T07:45:48.718558+0000","last_active":"2024-04-16T07:45:48.717426+0000","last_peered":"2024-04-16T07:45:48.717426+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.713866+0000","last_became_peered":"2024-04-16T07:45:48.713866+0000","last_unstale":"2024-04-16T07:45:48.717426+0000","last_undegraded":"2024-04-16T07:45:48.717426+0000","last_fullsized":"2024-04-16T07:45:48.717426+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.773933+0000","last_change":"2024-04-16T07:45:48.774974+0000","last_active":"2024-04-16T07:45:48.773933+0000","last_peered":"2024-04-16T07:45:48.773933+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.768713+0000","last_became_peered":"2024-04-16T07:45:48.768713+0000","last_unstale":"2024-04-16T07:45:48.773933+0000","last_undegraded":"2024-04-16T07:45:48.773933+0000","last_fullsized":"2024-04-16T07:45:48.773933+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:49.648519+0000","last_change":"2024-04-16T07:45:48.724954+0000","last_active":"2024-04-16T07:45:49.648519+0000","last_peered":"2024-04-16T07:45:49.648519+0000","last_clean":"2024-04-16T07:45:49.648519+0000","last_became_active":"2024-04-16T07:45:48.719985+0000","last_became_peered":"2024-04-16T07:45:48.719985+0000","last_unstale":"2024-04-16T07:45:49.648519+0000","last_undegraded":"2024-04-16T07:45:49.648519+0000","last_fullsized":"2024-04-16T07:45:49.648519+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.811632+0000","last_change":"2024-04-16T07:45:48.812462+0000","last_active":"2024-04-16T07:45:48.811632+0000","last_peered":"2024-04-16T07:45:48.811632+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.804250+0000","last_became_peered":"2024-04-16T07:45:48.804250+0000","last_unstale":"2024-04-16T07:45:48.811632+0000","last_undegraded":"2024-04-16T07:45:48.811632+0000","last_fullsized":"2024-04-16T07:45:48.811632+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:49.652389+0000","last_change":"2024-04-16T07:45:48.796564+0000","last_active":"2024-04-16T07:45:49.652389+0000","last_peered":"2024-04-16T07:45:49.652389+0000","last_clean":"2024-04-16T07:45:49.652389+0000","last_became_active":"2024-04-16T07:45:48.789023+0000","last_became_peered":"2024-04-16T07:45:48.789023+0000","last_unstale":"2024-04-16T07:45:49.652389+0000","last_undegraded":"2024-04-16T07:45:49.652389+0000","last_fullsized":"2024-04-16T07:45:49.652389+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:49.351146+0000","last_change":"2024-04-16T07:45:47.694188+0000","last_active":"2024-04-16T07:45:49.351146+0000","last_peered":"2024-04-16T07:45:49.351146+0000","last_clean":"2024-04-16T07:45:49.351146+0000","last_became_active":"2024-04-16T07:45:47.687698+0000","last_became_peered":"2024-04-16T07:45:47.687698+0000","last_unstale":"2024-04-16T07:45:49.351146+0000","last_undegraded":"2024-04-16T07:45:49.351146+0000","last_fullsized":"2024-04-16T07:45:49.351146+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_clean_scrub_stamp":"2024-04-16T07:45:46.626705+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:45:51.983 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:45:52.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:52.092+0000 7f0484264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:52.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:52.100+0000 7f0484264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:52.286 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:52.286 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-16T07:45:52.298 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-16T07:45:51.619113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107912,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379448,"statfs":{"total":386547056640,"available":386436554752,"internally_reserved":0,"allocated":593920,"data_stored":1870024,"data_compressed":89727,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.965235"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:49.652256+0000","last_change":"2024-04-16T07:45:48.769721+0000","last_active":"2024-04-16T07:45:49.652256+0000","last_peered":"2024-04-16T07:45:49.652256+0000","last_clean":"2024-04-16T07:45:49.652256+0000","last_became_active":"2024-04-16T07:45:48.764515+0000","last_became_peered":"2024-04-16T07:45:48.764515+0000","last_unstale":"2024-04-16T07:45:49.652256+0000","last_undegraded":"2024-04-16T07:45:49.652256+0000","last_fullsized":"2024-04-16T07:45:49.652256+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.765070+0000","last_change":"2024-04-16T07:45:48.766237+0000","last_active":"2024-04-16T07:45:48.765070+0000","last_peered":"2024-04-16T07:45:48.765070+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.761479+0000","last_became_peered":"2024-04-16T07:45:48.761479+0000","last_unstale":"2024-04-16T07:45:48.765070+0000","last_undegraded":"2024-04-16T07:45:48.765070+0000","last_fullsized":"2024-04-16T07:45:48.765070+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.765103+0000","last_change":"2024-04-16T07:45:48.766846+0000","last_active":"2024-04-16T07:45:48.765103+0000","last_peered":"2024-04-16T07:45:48.765103+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.761593+0000","last_became_peered":"2024-04-16T07:45:48.761593+0000","last_unstale":"2024-04-16T07:45:48.765103+0000","last_undegraded":"2024-04-16T07:45:48.765103+0000","last_fullsized":"2024-04-16T07:45:48.765103+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.717426+0000","last_change":"2024-04-16T07:45:48.718558+0000","last_active":"2024-04-16T07:45:48.717426+0000","last_peered":"2024-04-16T07:45:48.717426+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.713866+0000","last_became_peered":"2024-04-16T07:45:48.713866+0000","last_unstale":"2024-04-16T07:45:48.717426+0000","last_undegraded":"2024-04-16T07:45:48.717426+0000","last_fullsized":"2024-04-16T07:45:48.717426+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.773933+0000","last_change":"2024-04-16T07:45:48.774974+0000","last_active":"2024-04-16T07:45:48.773933+0000","last_peered":"2024-04-16T07:45:48.773933+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.768713+0000","last_became_peered":"2024-04-16T07:45:48.768713+0000","last_unstale":"2024-04-16T07:45:48.773933+0000","last_undegraded":"2024-04-16T07:45:48.773933+0000","last_fullsized":"2024-04-16T07:45:48.773933+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:49.648519+0000","last_change":"2024-04-16T07:45:48.724954+0000","last_active":"2024-04-16T07:45:49.648519+0000","last_peered":"2024-04-16T07:45:49.648519+0000","last_clean":"2024-04-16T07:45:49.648519+0000","last_became_active":"2024-04-16T07:45:48.719985+0000","last_became_peered":"2024-04-16T07:45:48.719985+0000","last_unstale":"2024-04-16T07:45:49.648519+0000","last_undegraded":"2024-04-16T07:45:49.648519+0000","last_fullsized":"2024-04-16T07:45:49.648519+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.811632+0000","last_change":"2024-04-16T07:45:48.812462+0000","last_active":"2024-04-16T07:45:48.811632+0000","last_peered":"2024-04-16T07:45:48.811632+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.804250+0000","last_became_peered":"2024-04-16T07:45:48.804250+0000","last_unstale":"2024-04-16T07:45:48.811632+0000","last_undegraded":"2024-04-16T07:45:48.811632+0000","last_fullsized":"2024-04-16T07:45:48.811632+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:49.652389+0000","last_change":"2024-04-16T07:45:48.796564+0000","last_active":"2024-04-16T07:45:49.652389+0000","last_peered":"2024-04-16T07:45:49.652389+0000","last_clean":"2024-04-16T07:45:49.652389+0000","last_became_active":"2024-04-16T07:45:48.789023+0000","last_became_peered":"2024-04-16T07:45:48.789023+0000","last_unstale":"2024-04-16T07:45:49.652389+0000","last_undegraded":"2024-04-16T07:45:49.652389+0000","last_fullsized":"2024-04-16T07:45:49.652389+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:49.351146+0000","last_change":"2024-04-16T07:45:47.694188+0000","last_active":"2024-04-16T07:45:49.351146+0000","last_peered":"2024-04-16T07:45:49.351146+0000","last_clean":"2024-04-16T07:45:49.351146+0000","last_became_active":"2024-04-16T07:45:47.687698+0000","last_became_peered":"2024-04-16T07:45:47.687698+0000","last_unstale":"2024-04-16T07:45:49.351146+0000","last_undegraded":"2024-04-16T07:45:49.351146+0000","last_fullsized":"2024-04-16T07:45:49.351146+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_clean_scrub_stamp":"2024-04-16T07:45:46.626705+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:45:52.299 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-16T07:45:52.299 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-16T07:45:52.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:52.407+0000 7f896ee2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:52.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:52.415+0000 7f896ee2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:52.605 INFO:teuthology.orchestra.run.smithi070.stdout:epoch 13 2024-04-16T07:45:52.605 INFO:teuthology.orchestra.run.smithi070.stdout:fsid e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6 2024-04-16T07:45:52.605 INFO:teuthology.orchestra.run.smithi070.stdout:created 2024-04-16T07:45:37.265201+0000 2024-04-16T07:45:52.605 INFO:teuthology.orchestra.run.smithi070.stdout:modified 2024-04-16T07:45:49.637253+0000 2024-04-16T07:45:52.605 INFO:teuthology.orchestra.run.smithi070.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-16T07:45:52.605 INFO:teuthology.orchestra.run.smithi070.stdout:crush_version 3 2024-04-16T07:45:52.605 INFO:teuthology.orchestra.run.smithi070.stdout:full_ratio 0.95 2024-04-16T07:45:52.605 INFO:teuthology.orchestra.run.smithi070.stdout:backfillfull_ratio 0.9 2024-04-16T07:45:52.605 INFO:teuthology.orchestra.run.smithi070.stdout:nearfull_ratio 0.85 2024-04-16T07:45:52.605 INFO:teuthology.orchestra.run.smithi070.stdout:require_min_compat_client luminous 2024-04-16T07:45:52.605 INFO:teuthology.orchestra.run.smithi070.stdout:min_compat_client jewel 2024-04-16T07:45:52.605 INFO:teuthology.orchestra.run.smithi070.stdout:require_osd_release squid 2024-04-16T07:45:52.605 INFO:teuthology.orchestra.run.smithi070.stdout:stretch_mode_enabled false 2024-04-16T07:45:52.606 INFO:teuthology.orchestra.run.smithi070.stdout:allow_crimson=true 2024-04-16T07:45:52.606 INFO:teuthology.orchestra.run.smithi070.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 12 flags hashpspool,nopgchange,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2024-04-16T07:45:52.606 INFO:teuthology.orchestra.run.smithi070.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 13 flags hashpspool,nopgchange,crimson stripe_width 0 application rbd read_balance_score 1.50 2024-04-16T07:45:52.606 INFO:teuthology.orchestra.run.smithi070.stdout:max_osd 4 2024-04-16T07:45:52.606 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.70:6801/1984645931 v2:172.21.15.70:6802/1984645931 exists,up 816d6f55-bf1d-48a9-9552-fc285b77358e 2024-04-16T07:45:52.606 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.70:6804/873227738 v2:172.21.15.70:6805/873227738 exists,up f777184a-794a-4817-ba31-872e243adcd4 2024-04-16T07:45:52.606 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.172:6804/911620790 v2:172.21.15.172:6805/911620790 exists,up 340d1b05-a919-45a7-bc75-1cb38943c8e2 2024-04-16T07:45:52.606 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.172:6808/1749870805 v2:172.21.15.172:6807/1749870805 exists,up 659133e9-2b60-43c8-9034-f4690c837fe9 2024-04-16T07:45:52.618 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.70:6801/1984645931 v2:172.21.15.70:6802/1984645931 exists,up 816d6f55-bf1d-48a9-9552-fc285b77358e', 'osd.1 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.70:6804/873227738 v2:172.21.15.70:6805/873227738 exists,up f777184a-794a-4817-ba31-872e243adcd4', 'osd.2 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.172:6804/911620790 v2:172.21.15.172:6805/911620790 exists,up 340d1b05-a919-45a7-bc75-1cb38943c8e2', 'osd.3 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.172:6808/1749870805 v2:172.21.15.172:6807/1749870805 exists,up 659133e9-2b60-43c8-9034-f4690c837fe9'] 2024-04-16T07:45:52.618 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-16T07:45:52.728 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-16T07:45:52.728 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mon.a", 2024-04-16T07:45:52.728 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster": "ceph", 2024-04-16T07:45:52.728 INFO:teuthology.orchestra.run.smithi070.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-16T07:45:52.728 INFO:teuthology.orchestra.run.smithi070.stdout: "admin_socket_mode": "", 2024-04-16T07:45:52.728 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-16T07:45:52.728 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_client_required": "cephx, none", 2024-04-16T07:45:52.728 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_cluster_required": "cephx", 2024-04-16T07:45:52.729 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_debug": "true", 2024-04-16T07:45:52.729 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-16T07:45:52.729 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-16T07:45:52.729 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_service_required": "cephx", 2024-04-16T07:45:52.729 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-16T07:45:52.729 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_supported": "cephx", 2024-04-16T07:45:52.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_aio": "true", 2024-04-16T07:45:52.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-16T07:45:52.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_aio_poll_ms": "250", 2024-04-16T07:45:52.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_aio_reap_max": "16", 2024-04-16T07:45:52.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_async_discard_threads": "0", 2024-04-16T07:45:52.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_block_size": "4096", 2024-04-16T07:45:52.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_debug_aio": "false", 2024-04-16T07:45:52.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-16T07:45:52.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-16T07:45:52.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_debug_inflight_ios": "false", 2024-04-16T07:45:52.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_enable_discard": "false", 2024-04-16T07:45:52.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_flock_retry": "3", 2024-04-16T07:45:52.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-16T07:45:52.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_inject_crash": "0", 2024-04-16T07:45:52.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-16T07:45:52.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_ioring": "false", 2024-04-16T07:45:52.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_ioring_hipri": "false", 2024-04-16T07:45:52.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-16T07:45:52.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-16T07:45:52.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-16T07:45:52.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-16T07:45:52.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_type": "", 2024-04-16T07:45:52.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_alloc_size": "1048576", 2024-04-16T07:45:52.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_allocator": "hybrid", 2024-04-16T07:45:52.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_buffered_io": "true", 2024-04-16T07:45:52.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_check_for_zeros": "false", 2024-04-16T07:45:52.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-16T07:45:52.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-16T07:45:52.731 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_compact_log_sync": "false", 2024-04-16T07:45:52.731 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-16T07:45:52.731 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-16T07:45:52.731 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-16T07:45:52.731 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-16T07:45:52.731 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_max_log_runway": "4194304", 2024-04-16T07:45:52.731 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_max_prefetch": "1048576", 2024-04-16T07:45:52.731 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_min_flush_size": "524288", 2024-04-16T07:45:52.731 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_min_log_runway": "1048576", 2024-04-16T07:45:52.731 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_replay_recovery": "false", 2024-04-16T07:45:52.731 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-16T07:45:52.731 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-16T07:45:52.731 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_sync_write": "false", 2024-04-16T07:45:52.731 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-16T07:45:52.731 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-16T07:45:52.731 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-16T07:45:52.732 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_allocation_from_file": "true", 2024-04-16T07:45:52.732 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_allocator": "hybrid", 2024-04-16T07:45:52.732 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-16T07:45:52.732 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-16T07:45:52.732 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-16T07:45:52.732 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-16T07:45:52.732 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-16T07:45:52.732 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-16T07:45:52.732 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-16T07:45:52.732 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_create": "true", 2024-04-16T07:45:52.732 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_db_create": "false", 2024-04-16T07:45:52.732 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_db_path": "", 2024-04-16T07:45:52.732 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_db_size": "0", 2024-04-16T07:45:52.732 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_path": "", 2024-04-16T07:45:52.732 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_preallocate_file": "false", 2024-04-16T07:45:52.732 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_size": "107374182400", 2024-04-16T07:45:52.733 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_wal_create": "false", 2024-04-16T07:45:52.733 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_wal_path": "", 2024-04-16T07:45:52.733 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_wal_size": "100663296", 2024-04-16T07:45:52.733 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_bluefs": "true", 2024-04-16T07:45:52.733 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-16T07:45:52.733 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-16T07:45:52.733 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-16T07:45:52.733 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-16T07:45:52.733 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-16T07:45:52.733 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-16T07:45:52.733 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-16T07:45:52.733 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-16T07:45:52.733 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_autotune": "true", 2024-04-16T07:45:52.733 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-16T07:45:52.733 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-16T07:45:52.733 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-16T07:45:52.734 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-16T07:45:52.734 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_size": "0", 2024-04-16T07:45:52.734 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-16T07:45:52.734 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-16T07:45:52.734 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-16T07:45:52.734 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-16T07:45:52.734 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_type": "2q", 2024-04-16T07:45:52.734 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-16T07:45:52.734 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_clone_cow": "true", 2024-04-16T07:45:52.734 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-16T07:45:52.734 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-16T07:45:52.734 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-16T07:45:52.734 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-16T07:45:52.734 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-16T07:45:52.734 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-16T07:45:52.734 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-16T07:45:52.735 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_compression_mode": "none", 2024-04-16T07:45:52.735 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-16T07:45:52.735 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_csum_type": "crc32c", 2024-04-16T07:45:52.735 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-04-16T07:45:52.735 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-16T07:45:52.735 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_freelist": "false", 2024-04-16T07:45:52.735 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-16T07:45:52.735 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-16T07:45:52.735 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-16T07:45:52.735 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-16T07:45:52.735 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-16T07:45:52.735 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-16T07:45:52.735 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-16T07:45:52.735 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-16T07:45:52.735 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-16T07:45:52.735 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-16T07:45:52.736 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-16T07:45:52.736 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-16T07:45:52.736 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-16T07:45:52.736 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_small_allocations": "0", 2024-04-16T07:45:52.736 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-16T07:45:52.736 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_default_buffered_read": "true", 2024-04-16T07:45:52.736 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_default_buffered_write": "false", 2024-04-16T07:45:52.736 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-16T07:45:52.736 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-16T07:45:52.736 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-16T07:45:52.736 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-16T07:45:52.736 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-16T07:45:52.736 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-16T07:45:52.736 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-16T07:45:52.736 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-16T07:45:52.736 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-16T07:45:52.737 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fail_eio": "false", 2024-04-16T07:45:52.737 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-16T07:45:52.737 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-16T07:45:52.737 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-16T07:45:52.737 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-16T07:45:52.737 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-16T07:45:52.737 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-16T07:45:52.737 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_on_mount": "false", 2024-04-16T07:45:52.737 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-16T07:45:52.737 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_on_umount": "false", 2024-04-16T07:45:52.737 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-16T07:45:52.737 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-16T07:45:52.737 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-16T07:45:52.738 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-16T07:45:52.738 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-16T07:45:52.738 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-16T07:45:52.738 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-16T07:45:52.738 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-16T07:45:52.738 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_ignore_data_csum": "false", 2024-04-16T07:45:52.738 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-16T07:45:52.738 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-16T07:45:52.738 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-16T07:45:52.738 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-16T07:45:52.738 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_log_op_age": "5.000000", 2024-04-16T07:45:52.738 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_max_alloc_size": "0", 2024-04-16T07:45:52.738 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_max_blob_size": "0", 2024-04-16T07:45:52.738 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-16T07:45:52.738 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-16T07:45:52.739 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-16T07:45:52.739 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_max_deferred_txc": "32", 2024-04-16T07:45:52.739 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_min_alloc_size": "0", 2024-04-16T07:45:52.739 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-16T07:45:52.739 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-16T07:45:52.739 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_nid_prealloc": "1024", 2024-04-16T07:45:52.739 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-16T07:45:52.739 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-16T07:45:52.739 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-16T07:45:52.739 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-16T07:45:52.739 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_retry_disk_reads": "3", 2024-04-16T07:45:52.739 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_rocksdb_cf": "true", 2024-04-16T07:45:52.739 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-16T07:45:52.739 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-16T07:45:52.739 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-16T07:45:52.739 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-16T07:45:52.739 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-16T07:45:52.740 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-16T07:45:52.740 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_spdk_mem": "512", 2024-04-16T07:45:52.740 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-16T07:45:52.740 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-16T07:45:52.740 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-16T07:45:52.740 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-16T07:45:52.740 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-16T07:45:52.740 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-16T07:45:52.740 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-16T07:45:52.740 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_tracing": "false", 2024-04-16T07:45:52.740 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-16T07:45:52.740 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-16T07:45:52.740 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-16T07:45:52.740 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-16T07:45:52.740 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-16T07:45:52.740 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-16T07:45:52.741 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-16T07:45:52.741 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-16T07:45:52.741 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-16T07:45:52.741 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_zero_block_detection": "false", 2024-04-16T07:45:52.741 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-16T07:45:52.741 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-16T07:45:52.741 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-16T07:45:52.741 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-16T07:45:52.741 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-16T07:45:52.741 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-16T07:45:52.741 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-16T07:45:52.741 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-16T07:45:52.741 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-16T07:45:52.741 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-16T07:45:52.742 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-16T07:45:52.742 INFO:teuthology.orchestra.run.smithi070.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-16T07:45:52.742 INFO:teuthology.orchestra.run.smithi070.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-16T07:45:52.742 INFO:teuthology.orchestra.run.smithi070.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-16T07:45:52.742 INFO:teuthology.orchestra.run.smithi070.stdout: "cephx_cluster_require_signatures": "false", 2024-04-16T07:45:52.742 INFO:teuthology.orchestra.run.smithi070.stdout: "cephx_cluster_require_version": "2", 2024-04-16T07:45:52.742 INFO:teuthology.orchestra.run.smithi070.stdout: "cephx_require_signatures": "false", 2024-04-16T07:45:52.742 INFO:teuthology.orchestra.run.smithi070.stdout: "cephx_require_version": "2", 2024-04-16T07:45:52.742 INFO:teuthology.orchestra.run.smithi070.stdout: "cephx_service_require_signatures": "false", 2024-04-16T07:45:52.742 INFO:teuthology.orchestra.run.smithi070.stdout: "cephx_service_require_version": "2", 2024-04-16T07:45:52.742 INFO:teuthology.orchestra.run.smithi070.stdout: "cephx_sign_messages": "true", 2024-04-16T07:45:52.742 INFO:teuthology.orchestra.run.smithi070.stdout: "chdir": "", 2024-04-16T07:45:52.742 INFO:teuthology.orchestra.run.smithi070.stdout: "client_acl_type": "", 2024-04-16T07:45:52.742 INFO:teuthology.orchestra.run.smithi070.stdout: "client_asio_thread_count": "2", 2024-04-16T07:45:52.742 INFO:teuthology.orchestra.run.smithi070.stdout: "client_cache_mid": "0.750000", 2024-04-16T07:45:52.742 INFO:teuthology.orchestra.run.smithi070.stdout: "client_cache_size": "16384", 2024-04-16T07:45:52.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_caps_release_delay": "5", 2024-04-16T07:45:52.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_check_pool_perm": "true", 2024-04-16T07:45:52.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-16T07:45:52.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_debug_force_sync_read": "false", 2024-04-16T07:45:52.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_debug_getattr_caps": "false", 2024-04-16T07:45:52.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_debug_inject_tick_delay": "0", 2024-04-16T07:45:52.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-16T07:45:52.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_die_on_failed_remount": "false", 2024-04-16T07:45:52.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_dirsize_rbytes": "true", 2024-04-16T07:45:52.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_force_lazyio": "false", 2024-04-16T07:45:52.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_fs": "", 2024-04-16T07:45:52.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-16T07:45:52.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_inject_release_failure": "false", 2024-04-16T07:45:52.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_max_inline_size": "4096", 2024-04-16T07:45:52.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-16T07:45:52.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_mds_namespace": "", 2024-04-16T07:45:52.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_metadata": "", 2024-04-16T07:45:52.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_mount_gid": "-1", 2024-04-16T07:45:52.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_mount_timeout": "300", 2024-04-16T07:45:52.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_mount_uid": "-1", 2024-04-16T07:45:52.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_mountpoint": "/", 2024-04-16T07:45:52.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_notify_timeout": "10", 2024-04-16T07:45:52.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_oc": "true", 2024-04-16T07:45:52.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_oc_max_dirty": "104857600", 2024-04-16T07:45:52.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-16T07:45:52.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_oc_max_objects": "1000", 2024-04-16T07:45:52.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_oc_size": "209715200", 2024-04-16T07:45:52.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_oc_target_dirty": "8388608", 2024-04-16T07:45:52.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_permissions": "true", 2024-04-16T07:45:52.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_quota": "true", 2024-04-16T07:45:52.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_quota_df": "true", 2024-04-16T07:45:52.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_readahead_max_bytes": "0", 2024-04-16T07:45:52.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_readahead_max_periods": "4", 2024-04-16T07:45:52.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_readahead_min": "131072", 2024-04-16T07:45:52.745 INFO:teuthology.orchestra.run.smithi070.stdout: "client_reconnect_stale": "false", 2024-04-16T07:45:52.745 INFO:teuthology.orchestra.run.smithi070.stdout: "client_shutdown_timeout": "30", 2024-04-16T07:45:52.745 INFO:teuthology.orchestra.run.smithi070.stdout: "client_snapdir": ".snap", 2024-04-16T07:45:52.745 INFO:teuthology.orchestra.run.smithi070.stdout: "client_tick_interval": "1", 2024-04-16T07:45:52.745 INFO:teuthology.orchestra.run.smithi070.stdout: "client_trace": "", 2024-04-16T07:45:52.745 INFO:teuthology.orchestra.run.smithi070.stdout: "client_try_dentry_invalidate": "false", 2024-04-16T07:45:52.745 INFO:teuthology.orchestra.run.smithi070.stdout: "client_use_faked_inos": "false", 2024-04-16T07:45:52.745 INFO:teuthology.orchestra.run.smithi070.stdout: "client_use_random_mds": "false", 2024-04-16T07:45:52.745 INFO:teuthology.orchestra.run.smithi070.stdout: "clog_to_graylog": "false", 2024-04-16T07:45:52.745 INFO:teuthology.orchestra.run.smithi070.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-16T07:45:52.745 INFO:teuthology.orchestra.run.smithi070.stdout: "clog_to_graylog_port": "12201", 2024-04-16T07:45:52.745 INFO:teuthology.orchestra.run.smithi070.stdout: "clog_to_monitors": "default=true", 2024-04-16T07:45:52.745 INFO:teuthology.orchestra.run.smithi070.stdout: "clog_to_syslog": "false", 2024-04-16T07:45:52.745 INFO:teuthology.orchestra.run.smithi070.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-16T07:45:52.745 INFO:teuthology.orchestra.run.smithi070.stdout: "clog_to_syslog_level": "info", 2024-04-16T07:45:52.745 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_addr": "-", 2024-04-16T07:45:52.745 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_network": "", 2024-04-16T07:45:52.746 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_network_interface": "", 2024-04-16T07:45:52.746 INFO:teuthology.orchestra.run.smithi070.stdout: "compressor_zlib_isal": "false", 2024-04-16T07:45:52.746 INFO:teuthology.orchestra.run.smithi070.stdout: "compressor_zlib_level": "5", 2024-04-16T07:45:52.746 INFO:teuthology.orchestra.run.smithi070.stdout: "compressor_zlib_winsize": "-15", 2024-04-16T07:45:52.746 INFO:teuthology.orchestra.run.smithi070.stdout: "compressor_zstd_level": "1", 2024-04-16T07:45:52.746 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-16T07:45:52.746 INFO:teuthology.orchestra.run.smithi070.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-16T07:45:52.746 INFO:teuthology.orchestra.run.smithi070.stdout: "crimson_alien_op_num_threads": "6", 2024-04-16T07:45:52.746 INFO:teuthology.orchestra.run.smithi070.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-16T07:45:52.746 INFO:teuthology.orchestra.run.smithi070.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-16T07:45:52.746 INFO:teuthology.orchestra.run.smithi070.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-16T07:45:52.746 INFO:teuthology.orchestra.run.smithi070.stdout: "crimson_osd_stat_interval": "0", 2024-04-16T07:45:52.746 INFO:teuthology.orchestra.run.smithi070.stdout: "crimson_seastar_cpu_cores": "", 2024-04-16T07:45:52.746 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_location": "", 2024-04-16T07:45:52.746 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_location_hook": "", 2024-04-16T07:45:52.746 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_location_hook_timeout": "10", 2024-04-16T07:45:52.747 INFO:teuthology.orchestra.run.smithi070.stdout: "daemonize": "false", 2024-04-16T07:45:52.747 INFO:teuthology.orchestra.run.smithi070.stdout: "daos_pool": "tank", 2024-04-16T07:45:52.747 INFO:teuthology.orchestra.run.smithi070.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-16T07:45:52.747 INFO:teuthology.orchestra.run.smithi070.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-16T07:45:52.747 INFO:teuthology.orchestra.run.smithi070.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-16T07:45:52.747 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_alienstore": "0/5", 2024-04-16T07:45:52.747 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_allow_any_pool_priority": "false", 2024-04-16T07:45:52.747 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_asok": "1/5", 2024-04-16T07:45:52.747 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_asok_assert_abort": "false", 2024-04-16T07:45:52.747 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_asserts_on_shutdown": "true", 2024-04-16T07:45:52.747 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_auth": "20/20", 2024-04-16T07:45:52.747 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_bdev": "1/3", 2024-04-16T07:45:52.747 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_bluefs": "1/5", 2024-04-16T07:45:52.747 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_bluestore": "1/5", 2024-04-16T07:45:52.747 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_buffer": "0/1", 2024-04-16T07:45:52.747 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_ceph_exporter": "1/5", 2024-04-16T07:45:52.747 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_cephfs_mirror": "0/5", 2024-04-16T07:45:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_cephsqlite": "0/5", 2024-04-16T07:45:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_client": "0/5", 2024-04-16T07:45:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_compressor": "1/5", 2024-04-16T07:45:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_context": "0/1", 2024-04-16T07:45:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_crimson_interrupt": "0/5", 2024-04-16T07:45:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_crush": "1/1", 2024-04-16T07:45:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_crypto": "1/5", 2024-04-16T07:45:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_cyanstore": "0/5", 2024-04-16T07:45:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_deliberately_leak_memory": "false", 2024-04-16T07:45:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_disable_randomized_ping": "false", 2024-04-16T07:45:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_dpdk": "1/5", 2024-04-16T07:45:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_eventtrace": "1/5", 2024-04-16T07:45:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_filer": "0/1", 2024-04-16T07:45:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_filestore": "1/3", 2024-04-16T07:45:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_finisher": "1/1", 2024-04-16T07:45:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_fuse": "1/5", 2024-04-16T07:45:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_heartbeat_testing_span": "0", 2024-04-16T07:45:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_heartbeatmap": "1/5", 2024-04-16T07:45:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-16T07:45:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_javaclient": "1/5", 2024-04-16T07:45:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_journal": "1/3", 2024-04-16T07:45:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_journaler": "0/5", 2024-04-16T07:45:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_kstore": "1/5", 2024-04-16T07:45:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_lockdep": "0/1", 2024-04-16T07:45:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mclock": "1/5", 2024-04-16T07:45:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mds": "1/5", 2024-04-16T07:45:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mds_balancer": "1/5", 2024-04-16T07:45:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mds_locker": "1/5", 2024-04-16T07:45:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mds_log": "1/5", 2024-04-16T07:45:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mds_log_expire": "1/5", 2024-04-16T07:45:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mds_migrator": "1/5", 2024-04-16T07:45:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mds_quiesce": "3/5", 2024-04-16T07:45:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_memstore": "1/5", 2024-04-16T07:45:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mgr": "2/5", 2024-04-16T07:45:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mgrc": "1/5", 2024-04-16T07:45:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mon": "20/20", 2024-04-16T07:45:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_monc": "0/10", 2024-04-16T07:45:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_ms": "1/1", 2024-04-16T07:45:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_none": "0/5", 2024-04-16T07:45:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_objclass": "0/5", 2024-04-16T07:45:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_objectcacher": "0/5", 2024-04-16T07:45:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_objecter": "0/1", 2024-04-16T07:45:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_optracker": "0/5", 2024-04-16T07:45:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_osd": "1/5", 2024-04-16T07:45:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_paxos": "20/20", 2024-04-16T07:45:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_perfcounter": "1/5", 2024-04-16T07:45:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_prioritycache": "1/5", 2024-04-16T07:45:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rados": "0/5", 2024-04-16T07:45:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rbd": "0/5", 2024-04-16T07:45:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rbd_mirror": "0/5", 2024-04-16T07:45:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rbd_pwl": "0/5", 2024-04-16T07:45:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rbd_replay": "0/5", 2024-04-16T07:45:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_refs": "0/0", 2024-04-16T07:45:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_reserver": "1/1", 2024-04-16T07:45:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rgw": "1/5", 2024-04-16T07:45:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rgw_access": "1/5", 2024-04-16T07:45:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rgw_datacache": "1/5", 2024-04-16T07:45:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rgw_dbstore": "1/5", 2024-04-16T07:45:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rgw_flight": "1/5", 2024-04-16T07:45:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-16T07:45:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rgw_notification": "1/5", 2024-04-16T07:45:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rgw_sync": "1/5", 2024-04-16T07:45:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rocksdb": "4/5", 2024-04-16T07:45:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore": "0/5", 2024-04-16T07:45:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_backref": "0/5", 2024-04-16T07:45:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_cache": "0/5", 2024-04-16T07:45:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_cleaner": "0/5", 2024-04-16T07:45:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_device": "0/5", 2024-04-16T07:45:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_epm": "0/5", 2024-04-16T07:45:52.752 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-16T07:45:52.752 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_journal": "0/5", 2024-04-16T07:45:52.752 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_lba": "0/5", 2024-04-16T07:45:52.752 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_odata": "0/5", 2024-04-16T07:45:52.752 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_omap": "0/5", 2024-04-16T07:45:52.752 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_onode": "0/5", 2024-04-16T07:45:52.752 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_t": "0/5", 2024-04-16T07:45:52.752 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_tm": "0/5", 2024-04-16T07:45:52.752 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_striper": "0/1", 2024-04-16T07:45:52.752 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_test": "0/5", 2024-04-16T07:45:52.752 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_throttle": "1/1", 2024-04-16T07:45:52.752 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_timer": "0/1", 2024-04-16T07:45:52.752 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_tp": "0/5", 2024-04-16T07:45:52.752 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_trace": "1/5", 2024-04-16T07:45:52.752 INFO:teuthology.orchestra.run.smithi070.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-16T07:45:52.752 INFO:teuthology.orchestra.run.smithi070.stdout: "device_failure_prediction_mode": "none", 2024-04-16T07:45:52.753 INFO:teuthology.orchestra.run.smithi070.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2024-04-16T07:45:52.753 INFO:teuthology.orchestra.run.smithi070.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-16T07:45:52.753 INFO:teuthology.orchestra.run.smithi070.stdout: "err_to_graylog": "false", 2024-04-16T07:45:52.753 INFO:teuthology.orchestra.run.smithi070.stdout: "err_to_journald": "false", 2024-04-16T07:45:52.753 INFO:teuthology.orchestra.run.smithi070.stdout: "err_to_stderr": "true", 2024-04-16T07:45:52.753 INFO:teuthology.orchestra.run.smithi070.stdout: "err_to_syslog": "false", 2024-04-16T07:45:52.753 INFO:teuthology.orchestra.run.smithi070.stdout: "event_tracing": "false", 2024-04-16T07:45:52.753 INFO:teuthology.orchestra.run.smithi070.stdout: "exporter_addr": "0.0.0.0", 2024-04-16T07:45:52.753 INFO:teuthology.orchestra.run.smithi070.stdout: "exporter_http_port": "9926", 2024-04-16T07:45:52.753 INFO:teuthology.orchestra.run.smithi070.stdout: "exporter_prio_limit": "5", 2024-04-16T07:45:52.753 INFO:teuthology.orchestra.run.smithi070.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-16T07:45:52.753 INFO:teuthology.orchestra.run.smithi070.stdout: "exporter_sort_metrics": "true", 2024-04-16T07:45:52.753 INFO:teuthology.orchestra.run.smithi070.stdout: "exporter_stats_period": "5", 2024-04-16T07:45:52.753 INFO:teuthology.orchestra.run.smithi070.stdout: "fake_statfs_for_testing": "0", 2024-04-16T07:45:52.753 INFO:teuthology.orchestra.run.smithi070.stdout: "fatal_signal_handlers": "true", 2024-04-16T07:45:52.753 INFO:teuthology.orchestra.run.smithi070.stdout: "filer_max_purge_ops": "10", 2024-04-16T07:45:52.753 INFO:teuthology.orchestra.run.smithi070.stdout: "filer_max_truncate_ops": "128", 2024-04-16T07:45:52.754 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_apply_finisher_threads": "1", 2024-04-16T07:45:52.754 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_blackhole": "false", 2024-04-16T07:45:52.754 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_btrfs_clone_range": "true", 2024-04-16T07:45:52.754 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_btrfs_snap": "true", 2024-04-16T07:45:52.754 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_caller_concurrency": "10", 2024-04-16T07:45:52.754 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_collect_device_partition_information": "true", 2024-04-16T07:45:52.754 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_commit_timeout": "600.000000", 2024-04-16T07:45:52.754 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_debug_inject_read_err": "false", 2024-04-16T07:45:52.754 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_debug_omap_check": "false", 2024-04-16T07:45:52.754 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_debug_verify_split": "false", 2024-04-16T07:45:52.754 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_dump_file": "", 2024-04-16T07:45:52.754 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-16T07:45:52.754 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-16T07:45:52.754 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_fadvise": "true", 2024-04-16T07:45:52.754 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_fail_eio": "true", 2024-04-16T07:45:52.754 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_fd_cache_shards": "16", 2024-04-16T07:45:52.754 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_fd_cache_size": "128", 2024-04-16T07:45:52.755 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_fiemap": "false", 2024-04-16T07:45:52.755 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_fiemap_threshold": "4096", 2024-04-16T07:45:52.755 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-16T07:45:52.755 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-16T07:45:52.755 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_inject_stall": "0", 2024-04-16T07:45:52.755 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_journal_parallel": "false", 2024-04-16T07:45:52.755 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_journal_trailing": "false", 2024-04-16T07:45:52.755 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_journal_writeahead": "false", 2024-04-16T07:45:52.755 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_kill_at": "0", 2024-04-16T07:45:52.755 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-16T07:45:52.755 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-16T07:45:52.755 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-16T07:45:52.755 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-16T07:45:52.755 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-16T07:45:52.755 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_inline_xattrs": "0", 2024-04-16T07:45:52.756 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-16T07:45:52.756 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-16T07:45:52.756 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-16T07:45:52.756 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-16T07:45:52.756 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_xattr_value_size": "0", 2024-04-16T07:45:52.756 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-16T07:45:52.756 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-16T07:45:52.756 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-16T07:45:52.756 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_merge_threshold": "-10", 2024-04-16T07:45:52.756 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-16T07:45:52.756 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_odsync_write": "false", 2024-04-16T07:45:52.756 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_omap_backend": "rocksdb", 2024-04-16T07:45:52.756 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_omap_backend_path": "", 2024-04-16T07:45:52.756 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-16T07:45:52.756 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-16T07:45:52.756 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-16T07:45:52.757 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_op_thread_timeout": "60", 2024-04-16T07:45:52.757 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_op_threads": "2", 2024-04-16T07:45:52.757 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_punch_hole": "false", 2024-04-16T07:45:52.757 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-16T07:45:52.757 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-16T07:45:52.757 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-16T07:45:52.757 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-16T07:45:52.757 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-16T07:45:52.757 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-16T07:45:52.757 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-16T07:45:52.757 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-16T07:45:52.757 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-16T07:45:52.757 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_max_ops": "50", 2024-04-16T07:45:52.757 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-16T07:45:52.758 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_seek_data_hole": "false", 2024-04-16T07:45:52.758 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_sloppy_crc": "false", 2024-04-16T07:45:52.758 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-16T07:45:52.758 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_splice": "false", 2024-04-16T07:45:52.758 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_split_multiple": "2", 2024-04-16T07:45:52.758 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_split_rand_factor": "20", 2024-04-16T07:45:52.758 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_update_to": "1000", 2024-04-16T07:45:52.758 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-16T07:45:52.758 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-16T07:45:52.758 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-16T07:45:52.758 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-16T07:45:52.758 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-16T07:45:52.758 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-16T07:45:52.758 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_enable": "true", 2024-04-16T07:45:52.758 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-16T07:45:52.758 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-16T07:45:52.758 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-16T07:45:52.759 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-16T07:45:52.759 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-16T07:45:52.759 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-16T07:45:52.759 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_xfs_extsize": "false", 2024-04-16T07:45:52.759 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_zfs_snap": "false", 2024-04-16T07:45:52.759 INFO:teuthology.orchestra.run.smithi070.stdout: "fio_dir": "/tmp/fio", 2024-04-16T07:45:52.759 INFO:teuthology.orchestra.run.smithi070.stdout: "fsid": "e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6", 2024-04-16T07:45:52.759 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_allow_other": "true", 2024-04-16T07:45:52.759 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_atomic_o_trunc": "true", 2024-04-16T07:45:52.759 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_big_writes": "true", 2024-04-16T07:45:52.759 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_debug": "false", 2024-04-16T07:45:52.759 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_default_permissions": "false", 2024-04-16T07:45:52.759 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_disable_pagecache": "false", 2024-04-16T07:45:52.759 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_max_write": "0", 2024-04-16T07:45:52.759 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_multithreaded": "true", 2024-04-16T07:45:52.759 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_require_active_mds": "true", 2024-04-16T07:45:52.759 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_set_user_groups": "true", 2024-04-16T07:45:52.760 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_splice_move": "true", 2024-04-16T07:45:52.760 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_splice_read": "true", 2024-04-16T07:45:52.760 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_splice_write": "true", 2024-04-16T07:45:52.760 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-16T07:45:52.760 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_use_invalidate_cb": "true", 2024-04-16T07:45:52.760 INFO:teuthology.orchestra.run.smithi070.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-16T07:45:52.760 INFO:teuthology.orchestra.run.smithi070.stdout: "gss_target_name": "ceph", 2024-04-16T07:45:52.760 INFO:teuthology.orchestra.run.smithi070.stdout: "heartbeat_file": "", 2024-04-16T07:45:52.760 INFO:teuthology.orchestra.run.smithi070.stdout: "heartbeat_inject_failure": "0", 2024-04-16T07:45:52.760 INFO:teuthology.orchestra.run.smithi070.stdout: "heartbeat_interval": "5", 2024-04-16T07:45:52.760 INFO:teuthology.orchestra.run.smithi070.stdout: "host": "smithi070", 2024-04-16T07:45:52.760 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-16T07:45:52.760 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-16T07:45:52.760 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-16T07:45:52.760 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-16T07:45:52.760 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-16T07:45:52.761 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-16T07:45:52.761 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-16T07:45:52.761 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-16T07:45:52.761 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-16T07:45:52.761 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-16T07:45:52.761 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-16T07:45:52.761 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-16T07:45:52.761 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-16T07:45:52.761 INFO:teuthology.orchestra.run.smithi070.stdout: "inject_early_sigterm": "false", 2024-04-16T07:45:52.761 INFO:teuthology.orchestra.run.smithi070.stdout: "jaeger_agent_port": "6799", 2024-04-16T07:45:52.761 INFO:teuthology.orchestra.run.smithi070.stdout: "jaeger_tracing_enable": "false", 2024-04-16T07:45:52.761 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_aio": "true", 2024-04-16T07:45:52.761 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_align_min_size": "65536", 2024-04-16T07:45:52.761 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_block_align": "true", 2024-04-16T07:45:52.761 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_block_size": "4096", 2024-04-16T07:45:52.761 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_dio": "true", 2024-04-16T07:45:52.761 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_discard": "false", 2024-04-16T07:45:52.761 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_force_aio": "false", 2024-04-16T07:45:52.762 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_ignore_corruption": "false", 2024-04-16T07:45:52.762 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_max_write_bytes": "10485760", 2024-04-16T07:45:52.762 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_max_write_entries": "100", 2024-04-16T07:45:52.762 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_replay_from": "0", 2024-04-16T07:45:52.762 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-16T07:45:52.762 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-16T07:45:52.762 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-16T07:45:52.762 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-16T07:45:52.762 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_write_header_frequency": "0", 2024-04-16T07:45:52.762 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_zero_on_create": "false", 2024-04-16T07:45:52.762 INFO:teuthology.orchestra.run.smithi070.stdout: "journaler_prefetch_periods": "10", 2024-04-16T07:45:52.762 INFO:teuthology.orchestra.run.smithi070.stdout: "journaler_prezero_periods": "5", 2024-04-16T07:45:52.762 INFO:teuthology.orchestra.run.smithi070.stdout: "journaler_write_head_interval": "15", 2024-04-16T07:45:52.762 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "", 2024-04-16T07:45:52.762 INFO:teuthology.orchestra.run.smithi070.stdout: "keyfile": "", 2024-04-16T07:45:52.762 INFO:teuthology.orchestra.run.smithi070.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-16T07:45:52.762 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_backend": "rocksdb", 2024-04-16T07:45:52.763 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_default_stripe_size": "65536", 2024-04-16T07:45:52.763 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_fsck_on_mount": "false", 2024-04-16T07:45:52.763 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-16T07:45:52.763 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_max_bytes": "67108864", 2024-04-16T07:45:52.763 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_max_ops": "512", 2024-04-16T07:45:52.763 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_nid_prealloc": "1024", 2024-04-16T07:45:52.763 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_onode_map_size": "1024", 2024-04-16T07:45:52.763 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-16T07:45:52.763 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_sync_submit_transaction": "false", 2024-04-16T07:45:52.763 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_sync_transaction": "false", 2024-04-16T07:45:52.763 INFO:teuthology.orchestra.run.smithi070.stdout: "librados_thread_count": "2", 2024-04-16T07:45:52.763 INFO:teuthology.orchestra.run.smithi070.stdout: "lockdep": "false", 2024-04-16T07:45:52.763 INFO:teuthology.orchestra.run.smithi070.stdout: "lockdep_force_backtrace": "false", 2024-04-16T07:45:52.763 INFO:teuthology.orchestra.run.smithi070.stdout: "log_coarse_timestamps": "true", 2024-04-16T07:45:52.763 INFO:teuthology.orchestra.run.smithi070.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-16T07:45:52.763 INFO:teuthology.orchestra.run.smithi070.stdout: "log_flush_on_exit": "false", 2024-04-16T07:45:52.763 INFO:teuthology.orchestra.run.smithi070.stdout: "log_graylog_host": "127.0.0.1", 2024-04-16T07:45:52.764 INFO:teuthology.orchestra.run.smithi070.stdout: "log_graylog_port": "12201", 2024-04-16T07:45:52.764 INFO:teuthology.orchestra.run.smithi070.stdout: "log_max_new": "1000", 2024-04-16T07:45:52.764 INFO:teuthology.orchestra.run.smithi070.stdout: "log_max_recent": "10000", 2024-04-16T07:45:52.764 INFO:teuthology.orchestra.run.smithi070.stdout: "log_stderr_prefix": "", 2024-04-16T07:45:52.764 INFO:teuthology.orchestra.run.smithi070.stdout: "log_stop_at_utilization": "0.970000", 2024-04-16T07:45:52.764 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": "true", 2024-04-16T07:45:52.764 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_graylog": "false", 2024-04-16T07:45:52.764 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_journald": "false", 2024-04-16T07:45:52.764 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_stderr": "false", 2024-04-16T07:45:52.764 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_syslog": "false", 2024-04-16T07:45:52.764 INFO:teuthology.orchestra.run.smithi070.stdout: "max_rotating_auth_attempts": "10", 2024-04-16T07:45:52.764 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-16T07:45:52.764 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_action_on_write_error": "1", 2024-04-16T07:45:52.764 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_alternate_name_max": "8192", 2024-04-16T07:45:52.764 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_asio_thread_count": "2", 2024-04-16T07:45:52.764 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_export_pin": "true", 2024-04-16T07:45:52.764 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_fragment_dirs": "true", 2024-04-16T07:45:52.765 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-16T07:45:52.765 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_fragment_interval": "5", 2024-04-16T07:45:52.765 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-16T07:45:52.765 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-16T07:45:52.765 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_interval": "10", 2024-04-16T07:45:52.765 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_max": "-1", 2024-04-16T07:45:52.765 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_max_until": "-1", 2024-04-16T07:45:52.765 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_merge_size": "50", 2024-04-16T07:45:52.765 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_midchunk": "0.300000", 2024-04-16T07:45:52.765 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-16T07:45:52.765 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_min_start": "0.200000", 2024-04-16T07:45:52.765 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_minchunk": "0.001000", 2024-04-16T07:45:52.765 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_mode": "0", 2024-04-16T07:45:52.765 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_need_max": "1.200000", 2024-04-16T07:45:52.765 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_need_min": "0.800000", 2024-04-16T07:45:52.765 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_overload_epochs": "2", 2024-04-16T07:45:52.766 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-16T07:45:52.766 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-16T07:45:52.766 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_split_bits": "3", 2024-04-16T07:45:52.766 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-16T07:45:52.766 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_split_size": "10000", 2024-04-16T07:45:52.766 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-16T07:45:52.766 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_target_decay": "10.000000", 2024-04-16T07:45:52.766 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-16T07:45:52.766 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_beacon_grace": "15.000000", 2024-04-16T07:45:52.766 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_beacon_interval": "4.000000", 2024-04-16T07:45:52.766 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-16T07:45:52.766 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-16T07:45:52.766 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_mid": "0.700000", 2024-04-16T07:45:52.766 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-16T07:45:52.766 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_quiesce_delay": "0", 2024-04-16T07:45:52.767 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-16T07:45:52.767 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-16T07:45:52.767 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-16T07:45:52.767 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_release_free_interval": "10", 2024-04-16T07:45:52.767 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_reservation": "0.050000", 2024-04-16T07:45:52.767 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-16T07:45:52.767 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_trim_interval": "1", 2024-04-16T07:45:52.767 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_trim_threshold": "262144", 2024-04-16T07:45:52.767 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-16T07:45:52.767 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-16T07:45:52.767 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-16T07:45:52.767 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_client_prealloc_inos": "1000", 2024-04-16T07:45:52.767 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-16T07:45:52.767 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_connect_bootstrapping": "false", 2024-04-16T07:45:52.767 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_damage_table_max_entries": "10000", 2024-04-16T07:45:52.767 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-16T07:45:52.767 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_debug_auth_pins": "false", 2024-04-16T07:45:52.767 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_debug_frag": "false", 2024-04-16T07:45:52.768 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_debug_scatterstat": "false", 2024-04-16T07:45:52.768 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_debug_subtrees": "false", 2024-04-16T07:45:52.768 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_decay_halflife": "5.000000", 2024-04-16T07:45:52.768 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_default_dir_hash": "2", 2024-04-16T07:45:52.768 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_defer_session_stale": "true", 2024-04-16T07:45:52.768 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_deny_all_reconnect": "false", 2024-04-16T07:45:52.768 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_dir_keys_per_op": "16384", 2024-04-16T07:45:52.768 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_dir_max_commit_size": "10", 2024-04-16T07:45:52.768 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_dir_max_entries": "0", 2024-04-16T07:45:52.768 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_dir_prefetch": "true", 2024-04-16T07:45:52.768 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-16T07:45:52.768 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-16T07:45:52.768 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_dump_cache_on_map": "false", 2024-04-16T07:45:52.768 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-16T07:45:52.768 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-16T07:45:52.768 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_early_reply": "true", 2024-04-16T07:45:52.768 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_enable_op_tracker": "true", 2024-04-16T07:45:52.769 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_enforce_unique_name": "true", 2024-04-16T07:45:52.769 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-16T07:45:52.769 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-16T07:45:52.769 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_export_ephemeral_random": "true", 2024-04-16T07:45:52.769 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-16T07:45:52.769 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-16T07:45:52.769 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-16T07:45:52.769 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-16T07:45:52.769 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-16T07:45:52.769 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-16T07:45:52.769 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-16T07:45:52.769 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-16T07:45:52.769 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_health_summarize_threshold": "10", 2024-04-16T07:45:52.769 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-16T07:45:52.769 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-16T07:45:52.769 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_inject_health_dummy": "false", 2024-04-16T07:45:52.770 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-16T07:45:52.770 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_inject_migrator_session_race": "false", 2024-04-16T07:45:52.770 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-16T07:45:52.770 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-16T07:45:52.770 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-16T07:45:52.770 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_join_fs": "", 2024-04-16T07:45:52.770 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_journal_format": "1", 2024-04-16T07:45:52.770 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_create_at": "0", 2024-04-16T07:45:52.770 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_export_at": "0", 2024-04-16T07:45:52.770 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_import_at": "0", 2024-04-16T07:45:52.770 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_journal_at": "0", 2024-04-16T07:45:52.770 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_journal_expire_at": "0", 2024-04-16T07:45:52.770 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_journal_replay_at": "0", 2024-04-16T07:45:52.770 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_link_at": "0", 2024-04-16T07:45:52.770 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_mdstable_at": "0", 2024-04-16T07:45:52.770 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_openc_at": "0", 2024-04-16T07:45:52.770 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_rename_at": "0", 2024-04-16T07:45:52.771 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_shutdown_at": "0", 2024-04-16T07:45:52.771 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-16T07:45:52.771 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_event_large_threshold": "524288", 2024-04-16T07:45:52.771 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_events_per_segment": "1024", 2024-04-16T07:45:52.771 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-16T07:45:52.771 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_max_events": "-1", 2024-04-16T07:45:52.771 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_max_segments": "128", 2024-04-16T07:45:52.771 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_pause": "false", 2024-04-16T07:45:52.771 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_segment_size": "0", 2024-04-16T07:45:52.771 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-16T07:45:52.771 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-16T07:45:52.771 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-16T07:45:52.771 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_trim_threshold": "128", 2024-04-16T07:45:52.771 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-16T07:45:52.771 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_warn_factor": "2.000000", 2024-04-16T07:45:52.771 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_caps_per_client": "1048576", 2024-04-16T07:45:52.771 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_completed_flushes": "100000", 2024-04-16T07:45:52.772 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_completed_requests": "100000", 2024-04-16T07:45:52.772 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_export_size": "20971520", 2024-04-16T07:45:52.772 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_file_recover": "32", 2024-04-16T07:45:52.772 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_purge_files": "64", 2024-04-16T07:45:52.772 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_purge_ops": "8192", 2024-04-16T07:45:52.772 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-16T07:45:52.772 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-16T07:45:52.772 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_snaps_per_dir": "100", 2024-04-16T07:45:52.772 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_metrics_update_interval": "2", 2024-04-16T07:45:52.772 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_min_caps_per_client": "100", 2024-04-16T07:45:52.772 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_min_caps_working_set": "10000", 2024-04-16T07:45:52.772 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-16T07:45:52.772 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_numa_node": "-1", 2024-04-16T07:45:52.772 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-16T07:45:52.772 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_op_complaint_time": "30.000000", 2024-04-16T07:45:52.772 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_op_history_duration": "600", 2024-04-16T07:45:52.772 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_op_history_size": "20", 2024-04-16T07:45:52.772 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_op_history_slow_op_size": "20", 2024-04-16T07:45:52.773 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-16T07:45:52.773 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_op_log_threshold": "5", 2024-04-16T07:45:52.773 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_ping_grace": "15", 2024-04-16T07:45:52.773 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_ping_interval": "5", 2024-04-16T07:45:52.773 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-16T07:45:52.773 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-16T07:45:52.773 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_recall_max_caps": "30000", 2024-04-16T07:45:52.773 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-16T07:45:52.773 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-16T07:45:52.773 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-16T07:45:52.773 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_recall_warning_threshold": "262144", 2024-04-16T07:45:52.773 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-16T07:45:52.773 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_replay_interval": "1.000000", 2024-04-16T07:45:52.773 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-16T07:45:52.773 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-16T07:45:52.773 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_root_ino_gid": "0", 2024-04-16T07:45:52.773 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_root_ino_uid": "0", 2024-04-16T07:45:52.774 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-16T07:45:52.774 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-16T07:45:52.774 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-16T07:45:52.774 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-16T07:45:52.774 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-16T07:45:52.774 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-16T07:45:52.774 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-16T07:45:52.774 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-16T07:45:52.774 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-16T07:45:52.774 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-16T07:45:52.774 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_shutdown_check": "0", 2024-04-16T07:45:52.774 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_skip_ino": "0", 2024-04-16T07:45:52.774 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-16T07:45:52.774 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_snap_max_uid": "4294967294", 2024-04-16T07:45:52.774 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_snap_min_uid": "0", 2024-04-16T07:45:52.774 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_snap_rstat": "false", 2024-04-16T07:45:52.774 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_standby_replay_damaged": "false", 2024-04-16T07:45:52.775 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_symlink_recovery": "true", 2024-04-16T07:45:52.775 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-16T07:45:52.775 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_thrash_exports": "0", 2024-04-16T07:45:52.775 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_thrash_fragments": "0", 2024-04-16T07:45:52.775 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_tick_interval": "5.000000", 2024-04-16T07:45:52.775 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_valgrind_exit": "false", 2024-04-16T07:45:52.775 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_verify_backtrace": "1", 2024-04-16T07:45:52.775 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_verify_scatter": "false", 2024-04-16T07:45:52.775 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-16T07:45:52.775 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_wipe_sessions": "false", 2024-04-16T07:45:52.775 INFO:teuthology.orchestra.run.smithi070.stdout: "mempool_debug": "false", 2024-04-16T07:45:52.775 INFO:teuthology.orchestra.run.smithi070.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-16T07:45:52.775 INFO:teuthology.orchestra.run.smithi070.stdout: "memstore_device_bytes": "1073741824", 2024-04-16T07:45:52.775 INFO:teuthology.orchestra.run.smithi070.stdout: "memstore_page_set": "false", 2024-04-16T07:45:52.775 INFO:teuthology.orchestra.run.smithi070.stdout: "memstore_page_size": "65536", 2024-04-16T07:45:52.775 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_client_bytes": "134217728", 2024-04-16T07:45:52.775 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_client_messages": "512", 2024-04-16T07:45:52.776 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-16T07:45:52.776 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-16T07:45:52.776 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-16T07:45:52.776 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-16T07:45:52.776 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_disabled_modules": "", 2024-04-16T07:45:52.776 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-16T07:45:52.776 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_max_pg_creating": "1024", 2024-04-16T07:45:52.776 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_max_pg_num_change": "128", 2024-04-16T07:45:52.776 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_mds_bytes": "134217728", 2024-04-16T07:45:52.776 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_mds_messages": "128", 2024-04-16T07:45:52.776 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-16T07:45:52.776 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_mon_bytes": "134217728", 2024-04-16T07:45:52.776 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_mon_messages": "128", 2024-04-16T07:45:52.776 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_osd_bytes": "536870912", 2024-04-16T07:45:52.776 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_osd_messages": "8192", 2024-04-16T07:45:52.776 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_pool": "true", 2024-04-16T07:45:52.776 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-16T07:45:52.777 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_standby_modules": "true", 2024-04-16T07:45:52.777 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_stats_period": "5", 2024-04-16T07:45:52.777 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_stats_threshold": "5", 2024-04-16T07:45:52.777 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_tick_period": "2", 2024-04-16T07:45:52.777 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-16T07:45:52.777 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-16T07:45:52.777 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_allow_pool_delete": "true", 2024-04-16T07:45:52.777 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_allow_pool_size_one": "true", 2024-04-16T07:45:52.777 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_auth_validate_all_caps": "true", 2024-04-16T07:45:52.777 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-16T07:45:52.777 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-16T07:45:52.777 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_bytes": "104857600", 2024-04-16T07:45:52.777 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_directed_command_retry": "2", 2024-04-16T07:45:52.777 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-16T07:45:52.777 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-16T07:45:52.777 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-16T07:45:52.778 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-16T07:45:52.778 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_hunt_parallel": "3", 2024-04-16T07:45:52.778 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_log_interval": "1.000000", 2024-04-16T07:45:52.778 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-16T07:45:52.778 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_ping_interval": "10.000000", 2024-04-16T07:45:52.778 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-16T07:45:52.778 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_target_rank": "-1", 2024-04-16T07:45:52.778 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-16T07:45:52.778 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-16T07:45:52.778 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-16T07:45:52.778 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_file_level": "debug", 2024-04-16T07:45:52.778 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_to_file": "true", 2024-04-16T07:45:52.778 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-16T07:45:52.778 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-16T07:45:52.778 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-16T07:45:52.779 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_to_journald": "false", 2024-04-16T07:45:52.779 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-16T07:45:52.779 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-16T07:45:52.779 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-16T07:45:52.779 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-04-16T07:45:52.779 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_compact_on_bootstrap": "false", 2024-04-16T07:45:52.779 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_compact_on_start": "false", 2024-04-16T07:45:52.779 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_compact_on_trim": "true", 2024-04-16T07:45:52.779 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-16T07:45:52.779 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-16T07:45:52.779 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-16T07:45:52.779 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cpu_threads": "4", 2024-04-16T07:45:52.779 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_crush_min_required_version": "hammer", 2024-04-16T07:45:52.779 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_daemon_bytes": "419430400", 2024-04-16T07:45:52.779 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-16T07:45:52.779 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_data_avail_crit": "5", 2024-04-16T07:45:52.780 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_data_avail_warn": "5", 2024-04-16T07:45:52.780 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_data_size_warn": "16106127360", 2024-04-16T07:45:52.780 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-16T07:45:52.780 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-16T07:45:52.780 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_dump_json": "false", 2024-04-16T07:45:52.780 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-16T07:45:52.780 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_dump_transactions": "false", 2024-04-16T07:45:52.780 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_extra_checks": "false", 2024-04-16T07:45:52.780 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-16T07:45:52.780 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-16T07:45:52.780 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_no_require_reef": "false", 2024-04-16T07:45:52.780 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_no_require_squid": "false", 2024-04-16T07:45:52.780 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-16T07:45:52.780 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-16T07:45:52.780 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-16T07:45:52.781 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_down_mkfs_grace": "120", 2024-04-16T07:45:52.781 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_down_uptime_grace": "60", 2024-04-16T07:45:52.781 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_election_default_strategy": "1", 2024-04-16T07:45:52.781 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_election_timeout": "5.000000", 2024-04-16T07:45:52.781 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-16T07:45:52.781 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_elector_ping_divisor": "2", 2024-04-16T07:45:52.781 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-16T07:45:52.781 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_enable_op_tracker": "true", 2024-04-16T07:45:52.781 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_fake_pool_delete": "false", 2024-04-16T07:45:52.781 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_force_quorum_join": "false", 2024-04-16T07:45:52.781 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_globalid_prealloc": "10000", 2024-04-16T07:45:52.782 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_health_detail_to_clog": "false", 2024-04-16T07:45:52.782 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_health_log_update_period": "5", 2024-04-16T07:45:52.782 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_health_max_detail": "50", 2024-04-16T07:45:52.782 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_health_to_clog": "true", 2024-04-16T07:45:52.782 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_health_to_clog_interval": "600", 2024-04-16T07:45:52.782 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-16T07:45:52.782 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_host": "172.21.15.70", 2024-04-16T07:45:52.782 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_host_override": "", 2024-04-16T07:45:52.782 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_initial_members": "", 2024-04-16T07:45:52.783 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-16T07:45:52.783 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-16T07:45:52.783 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-16T07:45:52.783 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-16T07:45:52.783 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-16T07:45:52.783 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_lease": "5.000000", 2024-04-16T07:45:52.783 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-16T07:45:52.783 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-16T07:45:52.783 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_log_full_interval": "50", 2024-04-16T07:45:52.783 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_log_max": "10000", 2024-04-16T07:45:52.783 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_log_max_summary": "50", 2024-04-16T07:45:52.783 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-16T07:45:52.783 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_max_log_epochs": "500", 2024-04-16T07:45:52.783 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-16T07:45:52.783 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-16T07:45:52.783 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_max_osd": "10000", 2024-04-16T07:45:52.784 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_max_pg_per_osd": "10000", 2024-04-16T07:45:52.784 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_max_pool_pg_num": "65536", 2024-04-16T07:45:52.784 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-16T07:45:52.784 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-16T07:45:52.784 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_mds_force_trim_to": "0", 2024-04-16T07:45:52.784 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_mds_skip_sanity": "false", 2024-04-16T07:45:52.784 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_memory_autotune": "true", 2024-04-16T07:45:52.784 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_memory_target": "2147483648", 2024-04-16T07:45:52.784 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_mgr_beacon_grace": "30", 2024-04-16T07:45:52.784 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-16T07:45:52.784 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_mgr_digest_period": "5", 2024-04-16T07:45:52.784 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_mgr_inactive_grace": "60", 2024-04-16T07:45:52.784 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-16T07:45:52.784 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-16T07:45:52.784 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_min_osdmap_epochs": "50", 2024-04-16T07:45:52.784 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_op_complaint_time": "30", 2024-04-16T07:45:52.784 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_op_history_duration": "600", 2024-04-16T07:45:52.785 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_op_history_size": "20", 2024-04-16T07:45:52.785 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_op_history_slow_op_size": "20", 2024-04-16T07:45:52.785 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-16T07:45:52.785 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_op_log_threshold": "5", 2024-04-16T07:45:52.785 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-16T07:45:52.785 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-16T07:45:52.785 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-16T07:45:52.785 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_auto_mark_in": "false", 2024-04-16T07:45:52.785 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-16T07:45:52.785 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-16T07:45:52.785 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-16T07:45:52.785 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_cache_size": "500", 2024-04-16T07:45:52.785 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-16T07:45:52.785 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-16T07:45:52.785 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-16T07:45:52.785 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_down_out_interval": "600", 2024-04-16T07:45:52.786 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-16T07:45:52.786 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-16T07:45:52.786 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_force_trim_to": "0", 2024-04-16T07:45:52.786 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-16T07:45:52.786 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-16T07:45:52.786 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-16T07:45:52.786 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-16T07:45:52.786 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-16T07:45:52.786 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-16T07:45:52.786 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-16T07:45:52.786 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_min_down_reporters": "2", 2024-04-16T07:45:52.786 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-16T07:45:52.786 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-16T07:45:52.786 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-16T07:45:52.786 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-16T07:45:52.786 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-16T07:45:52.787 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-16T07:45:52.787 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_report_timeout": "900", 2024-04-16T07:45:52.787 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-16T07:45:52.787 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-16T07:45:52.787 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-16T07:45:52.787 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-16T07:45:52.787 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-16T07:45:52.787 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-16T07:45:52.787 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-16T07:45:52.787 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-16T07:45:52.787 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-16T07:45:52.787 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_pg_stuck_threshold": "60", 2024-04-16T07:45:52.787 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-16T07:45:52.787 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-16T07:45:52.787 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-16T07:45:52.787 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-16T07:45:52.787 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-16T07:45:52.788 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-16T07:45:52.788 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_probe_timeout": "2.000000", 2024-04-16T07:45:52.788 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_reweight_max_change": "0.050000", 2024-04-16T07:45:52.788 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_reweight_max_osds": "4", 2024-04-16T07:45:52.788 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-16T07:45:52.788 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-16T07:45:52.788 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-16T07:45:52.788 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-16T07:45:52.788 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-16T07:45:52.788 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_scrub_interval": "86400", 2024-04-16T07:45:52.788 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_scrub_max_keys": "100", 2024-04-16T07:45:52.788 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_scrub_timeout": "300", 2024-04-16T07:45:52.788 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_session_timeout": "300", 2024-04-16T07:45:52.788 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_smart_report_timeout": "5", 2024-04-16T07:45:52.788 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_stat_smooth_intervals": "6", 2024-04-16T07:45:52.788 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-16T07:45:52.788 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_stretch_pool_min_size": "2", 2024-04-16T07:45:52.789 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_stretch_pool_size": "4", 2024-04-16T07:45:52.789 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-16T07:45:52.789 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-16T07:45:52.789 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_sync_debug": "false", 2024-04-16T07:45:52.789 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-16T07:45:52.789 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-16T07:45:52.789 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_sync_provider_kill_at": "0", 2024-04-16T07:45:52.789 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_sync_requester_kill_at": "0", 2024-04-16T07:45:52.789 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_sync_timeout": "60.000000", 2024-04-16T07:45:52.789 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_target_pg_per_osd": "100", 2024-04-16T07:45:52.789 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_tick_interval": "5", 2024-04-16T07:45:52.789 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_timecheck_interval": "300.000000", 2024-04-16T07:45:52.789 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-16T07:45:52.789 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_use_min_delay_socket": "false", 2024-04-16T07:45:52.789 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_older_version_delay": "604800", 2024-04-16T07:45:52.789 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-16T07:45:52.789 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-16T07:45:52.790 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-16T07:45:52.790 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-16T07:45:52.790 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-16T07:45:52.790 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-16T07:45:52.790 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-16T07:45:52.790 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_misplaced": "false", 2024-04-16T07:45:52.790 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-16T07:45:52.790 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_older_version": "true", 2024-04-16T07:45:52.790 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-16T07:45:52.790 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-16T07:45:52.790 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-16T07:45:52.790 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-16T07:45:52.790 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-16T07:45:52.790 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-16T07:45:52.790 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-16T07:45:52.790 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-16T07:45:52.791 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-16T07:45:52.791 INFO:teuthology.orchestra.run.smithi070.stdout: "monmap": "", 2024-04-16T07:45:52.791 INFO:teuthology.orchestra.run.smithi070.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-16T07:45:52.791 INFO:teuthology.orchestra.run.smithi070.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-16T07:45:52.791 INFO:teuthology.orchestra.run.smithi070.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-16T07:45:52.791 INFO:teuthology.orchestra.run.smithi070.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-16T07:45:52.791 INFO:teuthology.orchestra.run.smithi070.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-16T07:45:52.791 INFO:teuthology.orchestra.run.smithi070.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-16T07:45:52.791 INFO:teuthology.orchestra.run.smithi070.stdout: "motr_tracing_enabled": "false", 2024-04-16T07:45:52.791 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_op_threads": "3", 2024-04-16T07:45:52.791 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-16T07:45:52.791 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_cm": "false", 2024-04-16T07:45:52.791 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_device_name": "", 2024-04-16T07:45:52.791 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_dscp": "96", 2024-04-16T07:45:52.791 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-16T07:45:52.791 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-16T07:45:52.791 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_local_gid": "", 2024-04-16T07:45:52.792 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-16T07:45:52.792 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_port_num": "1", 2024-04-16T07:45:52.792 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-16T07:45:52.792 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-16T07:45:52.792 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-16T07:45:52.792 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-16T07:45:52.792 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_sl": "3", 2024-04-16T07:45:52.792 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_support_srq": "true", 2024-04-16T07:45:52.792 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_type": "ib", 2024-04-16T07:45:52.792 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_reap_threshold": "5", 2024-04-16T07:45:52.792 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_before_connect": "false", 2024-04-16T07:45:52.792 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_ipv4": "true", 2024-04-16T07:45:52.792 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_ipv6": "false", 2024-04-16T07:45:52.792 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_msgr1": "true", 2024-04-16T07:45:52.792 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_msgr2": "true", 2024-04-16T07:45:52.792 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_port_max": "7568", 2024-04-16T07:45:52.793 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_port_min": "6800", 2024-04-16T07:45:52.793 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-16T07:45:52.793 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_retry_count": "3", 2024-04-16T07:45:52.793 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_retry_delay": "5", 2024-04-16T07:45:52.793 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_blackhole_client": "false", 2024-04-16T07:45:52.793 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_blackhole_mds": "false", 2024-04-16T07:45:52.793 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_blackhole_mgr": "false", 2024-04-16T07:45:52.793 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_blackhole_mon": "false", 2024-04-16T07:45:52.793 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_blackhole_osd": "false", 2024-04-16T07:45:52.793 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_client_mode": "crc", 2024-04-16T07:45:52.793 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_cluster_mode": "crc", 2024-04-16T07:45:52.793 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_cluster_type": "", 2024-04-16T07:45:52.793 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_compress_secure": "false", 2024-04-16T07:45:52.793 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_connection_idle_timeout": "900", 2024-04-16T07:45:52.793 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_connection_ready_timeout": "10", 2024-04-16T07:45:52.793 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_crc_data": "true", 2024-04-16T07:45:52.793 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_crc_header": "true", 2024-04-16T07:45:52.794 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_die_on_bad_msg": "false", 2024-04-16T07:45:52.794 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_die_on_bug": "true", 2024-04-16T07:45:52.794 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_die_on_old_message": "true", 2024-04-16T07:45:52.794 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_die_on_skipped_message": "false", 2024-04-16T07:45:52.794 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-16T07:45:52.794 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-16T07:45:52.794 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_coremask": "0xF", 2024-04-16T07:45:52.794 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-16T07:45:52.794 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-16T07:45:52.794 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_enable_tso": "true", 2024-04-16T07:45:52.794 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-16T07:45:52.794 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-16T07:45:52.794 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_hugepages": "", 2024-04-16T07:45:52.794 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-16T07:45:52.794 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-16T07:45:52.794 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_lro": "true", 2024-04-16T07:45:52.795 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_memory_channel": "4", 2024-04-16T07:45:52.795 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-16T07:45:52.795 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_pmd": "", 2024-04-16T07:45:52.795 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_port_id": "0", 2024-04-16T07:45:52.795 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-16T07:45:52.795 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-16T07:45:52.795 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dump_on_send": "false", 2024-04-16T07:45:52.795 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_initial_backoff": "0.200000", 2024-04-16T07:45:52.795 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_inject_delay_max": "1.000000", 2024-04-16T07:45:52.795 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-16T07:45:52.795 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_inject_delay_type": "", 2024-04-16T07:45:52.795 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-16T07:45:52.795 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_inject_network_congestion": "0", 2024-04-16T07:45:52.795 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_inject_socket_failures": "0", 2024-04-16T07:45:52.795 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_learn_addr_from_peer": "true", 2024-04-16T07:45:52.795 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_max_accept_failures": "4", 2024-04-16T07:45:52.796 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_max_backoff": "15.000000", 2024-04-16T07:45:52.796 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_mon_client_mode": "crc", 2024-04-16T07:45:52.796 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_mon_cluster_mode": "crc", 2024-04-16T07:45:52.796 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_mon_service_mode": "crc", 2024-04-16T07:45:52.796 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_osd_compress_min_size": "1024", 2024-04-16T07:45:52.796 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_osd_compress_mode": "none", 2024-04-16T07:45:52.796 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-16T07:45:52.796 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-16T07:45:52.796 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_pq_min_cost": "65536", 2024-04-16T07:45:52.796 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_public_type": "", 2024-04-16T07:45:52.796 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_service_mode": "crc", 2024-04-16T07:45:52.796 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_tcp_listen_backlog": "512", 2024-04-16T07:45:52.796 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_tcp_nodelay": "true", 2024-04-16T07:45:52.796 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-16T07:45:52.796 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_tcp_rcvbuf": "0", 2024-04-16T07:45:52.796 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_type": "async+posix", 2024-04-16T07:45:52.796 INFO:teuthology.orchestra.run.smithi070.stdout: "no_config_file": "false", 2024-04-16T07:45:52.796 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter_completion_locks_per_session": "32", 2024-04-16T07:45:52.797 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-16T07:45:52.797 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-16T07:45:52.797 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter_inflight_ops": "1024", 2024-04-16T07:45:52.797 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-16T07:45:52.797 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-16T07:45:52.797 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter_tick_interval": "5.000000", 2024-04-16T07:45:52.797 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter_timeout": "10.000000", 2024-04-16T07:45:52.797 INFO:teuthology.orchestra.run.smithi070.stdout: "objectstore_blackhole": "false", 2024-04-16T07:45:52.797 INFO:teuthology.orchestra.run.smithi070.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-16T07:45:52.797 INFO:teuthology.orchestra.run.smithi070.stdout: "openssl_engine_opts": "", 2024-04-16T07:45:52.797 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_agent_delay_time": "5.000000", 2024-04-16T07:45:52.797 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_agent_hist_halflife": "1000", 2024-04-16T07:45:52.797 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_agent_max_low_ops": "2", 2024-04-16T07:45:52.797 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_agent_max_ops": "4", 2024-04-16T07:45:52.797 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-16T07:45:52.797 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-16T07:45:52.797 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_agent_slop": "0.020000", 2024-04-16T07:45:52.798 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-16T07:45:52.798 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-16T07:45:52.798 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_asio_thread_count": "2", 2024-04-16T07:45:52.798 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_async_recovery_min_cost": "100", 2024-04-16T07:45:52.798 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-16T07:45:52.798 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-16T07:45:52.798 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_backfill_scan_max": "512", 2024-04-16T07:45:52.798 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_backfill_scan_min": "64", 2024-04-16T07:45:52.798 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_backoff_on_degraded": "false", 2024-04-16T07:45:52.798 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_backoff_on_peering": "false", 2024-04-16T07:45:52.798 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_backoff_on_unfound": "true", 2024-04-16T07:45:52.798 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_beacon_report_interval": "300", 2024-04-16T07:45:52.798 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bench_duration": "30", 2024-04-16T07:45:52.799 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-16T07:45:52.799 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bench_max_block_size": "67108864", 2024-04-16T07:45:52.799 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-16T07:45:52.799 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_blkin_trace_all": "false", 2024-04-16T07:45:52.799 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-16T07:45:52.799 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-16T07:45:52.799 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-16T07:45:52.799 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-16T07:45:52.799 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_check_for_log_corruption": "false", 2024-04-16T07:45:52.799 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-16T07:45:52.799 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-16T07:45:52.799 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-16T07:45:52.799 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-16T07:45:52.799 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_class_update_on_start": "true", 2024-04-16T07:45:52.799 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_client_message_cap": "256", 2024-04-16T07:45:52.799 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_client_message_size_cap": "524288000", 2024-04-16T07:45:52.799 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_client_op_priority": "63", 2024-04-16T07:45:52.800 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_client_watch_timeout": "30", 2024-04-16T07:45:52.800 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_command_max_records": "256", 2024-04-16T07:45:52.800 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_compact_on_start": "false", 2024-04-16T07:45:52.800 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-16T07:45:52.800 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-16T07:45:52.800 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-16T07:45:52.800 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_crush_update_on_start": "true", 2024-04-16T07:45:52.800 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_crush_update_weight_set": "true", 2024-04-16T07:45:52.800 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-16T07:45:52.800 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-16T07:45:52.800 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-16T07:45:52.800 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-16T07:45:52.800 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-16T07:45:52.800 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_feed_pullee": "-1", 2024-04-16T07:45:52.800 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-16T07:45:52.800 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-16T07:45:52.800 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-16T07:45:52.801 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_misdirected_ops": "false", 2024-04-16T07:45:52.801 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_no_acting_change": "false", 2024-04-16T07:45:52.801 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_no_purge_strays": "false", 2024-04-16T07:45:52.801 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_op_order": "false", 2024-04-16T07:45:52.801 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-16T07:45:52.801 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-16T07:45:52.801 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-16T07:45:52.801 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-16T07:45:52.801 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_shutdown": "false", 2024-04-16T07:45:52.801 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-16T07:45:52.801 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-16T07:45:52.801 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_trim_objects": "false", 2024-04-16T07:45:52.801 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-16T07:45:52.802 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-16T07:45:52.802 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_verify_snaps": "false", 2024-04-16T07:45:52.802 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-16T07:45:52.802 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-16T07:45:52.802 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_deep_scrub_keys": "1024", 2024-04-16T07:45:52.802 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-16T07:45:52.802 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-16T07:45:52.802 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-16T07:45:52.802 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_deep_scrub_stride": "524288", 2024-04-16T07:45:52.802 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-16T07:45:52.802 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-16T07:45:52.802 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_default_notify_timeout": "30", 2024-04-16T07:45:52.802 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_delete_sleep": "0.000000", 2024-04-16T07:45:52.802 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-16T07:45:52.802 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-16T07:45:52.803 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-16T07:45:52.803 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_discard_disconnected_ops": "true", 2024-04-16T07:45:52.803 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_enable_op_tracker": "true", 2024-04-16T07:45:52.803 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-16T07:45:52.803 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-16T07:45:52.803 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-16T07:45:52.803 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-16T07:45:52.803 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_fast_info": "true", 2024-04-16T07:45:52.803 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_fast_shutdown": "true", 2024-04-16T07:45:52.803 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-16T07:45:52.803 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-16T07:45:52.803 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-16T07:45:52.803 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-16T07:45:52.803 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-16T07:45:52.803 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_function_tracing": "false", 2024-04-16T07:45:52.803 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_heartbeat_grace": "20", 2024-04-16T07:45:52.804 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_heartbeat_interval": "6", 2024-04-16T07:45:52.804 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-16T07:45:52.804 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_heartbeat_min_peers": "10", 2024-04-16T07:45:52.804 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_heartbeat_min_size": "2000", 2024-04-16T07:45:52.804 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_heartbeat_stale": "600", 2024-04-16T07:45:52.804 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-16T07:45:52.804 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_hit_set_max_size": "100000", 2024-04-16T07:45:52.804 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_hit_set_min_size": "1000", 2024-04-16T07:45:52.804 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-16T07:45:52.804 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-16T07:45:52.804 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-16T07:45:52.804 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-16T07:45:52.804 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-16T07:45:52.804 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-16T07:45:52.804 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_journal_size": "5120", 2024-04-16T07:45:52.804 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_kill_backfill_at": "0", 2024-04-16T07:45:52.804 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-16T07:45:52.805 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_map_cache_size": "50", 2024-04-16T07:45:52.805 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_map_dedup": "true", 2024-04-16T07:45:52.805 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_map_message_max": "40", 2024-04-16T07:45:52.805 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-16T07:45:52.805 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_map_share_max_epochs": "40", 2024-04-16T07:45:52.805 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_attr_name_len": "100", 2024-04-16T07:45:52.805 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_attr_size": "0", 2024-04-16T07:45:52.805 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_backfills": "1", 2024-04-16T07:45:52.805 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_markdown_count": "5", 2024-04-16T07:45:52.805 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_markdown_period": "600", 2024-04-16T07:45:52.805 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_object_name_len": "2048", 2024-04-16T07:45:52.805 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_object_namespace_len": "256", 2024-04-16T07:45:52.805 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_object_size": "134217728", 2024-04-16T07:45:52.805 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-16T07:45:52.805 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-16T07:45:52.805 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_pg_blocked_by": "16", 2024-04-16T07:45:52.805 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_pg_log_entries": "10000", 2024-04-16T07:45:52.806 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-16T07:45:52.806 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_pgls": "1024", 2024-04-16T07:45:52.806 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_push_cost": "8388608", 2024-04-16T07:45:52.806 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_push_objects": "10", 2024-04-16T07:45:52.806 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_scrubs": "3", 2024-04-16T07:45:52.806 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-16T07:45:52.806 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_trimming_pgs": "2", 2024-04-16T07:45:52.806 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_write_op_reply_len": "64", 2024-04-16T07:45:52.806 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_write_size": "90", 2024-04-16T07:45:52.806 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-16T07:45:52.806 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-16T07:45:52.806 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-16T07:45:52.806 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-16T07:45:52.806 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-16T07:45:52.806 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-16T07:45:52.807 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-16T07:45:52.807 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-16T07:45:52.807 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_profile": "balanced", 2024-04-16T07:45:52.807 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-16T07:45:52.807 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-16T07:45:52.807 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-16T07:45:52.807 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-16T07:45:52.807 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-16T07:45:52.807 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-16T07:45:52.807 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-16T07:45:52.807 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-16T07:45:52.807 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-16T07:45:52.807 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-16T07:45:52.807 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-16T07:45:52.807 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_memory_base": "805306368", 2024-04-16T07:45:52.807 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_memory_cache_min": "134217728", 2024-04-16T07:45:52.807 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-16T07:45:52.808 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-16T07:45:52.808 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_memory_target": "4294967296", 2024-04-16T07:45:52.808 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_memory_target_autotune": "false", 2024-04-16T07:45:52.808 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-16T07:45:52.808 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_min_pg_log_entries": "250", 2024-04-16T07:45:52.808 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_min_recovery_priority": "0", 2024-04-16T07:45:52.808 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-16T07:45:52.808 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-16T07:45:52.808 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mon_report_interval": "5", 2024-04-16T07:45:52.808 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-16T07:45:52.808 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-16T07:45:52.808 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_num_cache_shards": "32", 2024-04-16T07:45:52.808 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_num_op_tracker_shard": "32", 2024-04-16T07:45:52.808 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_numa_auto_affinity": "true", 2024-04-16T07:45:52.808 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_numa_node": "-1", 2024-04-16T07:45:52.808 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_numa_prefer_iface": "true", 2024-04-16T07:45:52.808 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-16T07:45:52.808 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_objecter_finishers": "1", 2024-04-16T07:45:52.809 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_objectstore": "bluestore", 2024-04-16T07:45:52.809 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_objectstore_fuse": "false", 2024-04-16T07:45:52.809 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_objectstore_tracing": "false", 2024-04-16T07:45:52.809 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_complaint_time": "30.000000", 2024-04-16T07:45:52.809 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_history_duration": "600", 2024-04-16T07:45:52.809 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_history_size": "20", 2024-04-16T07:45:52.809 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_history_slow_op_size": "20", 2024-04-16T07:45:52.809 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-16T07:45:52.809 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_log_threshold": "5", 2024-04-16T07:45:52.809 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_num_shards": "0", 2024-04-16T07:45:52.809 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_num_shards_hdd": "5", 2024-04-16T07:45:52.809 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_num_shards_ssd": "8", 2024-04-16T07:45:52.809 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-16T07:45:52.809 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-16T07:45:52.809 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-16T07:45:52.809 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-16T07:45:52.810 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_pq_min_cost": "65536", 2024-04-16T07:45:52.810 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-16T07:45:52.810 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_queue_cut_off": "high", 2024-04-16T07:45:52.810 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-16T07:45:52.810 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_thread_timeout": "15", 2024-04-16T07:45:52.810 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_open_classes_on_start": "true", 2024-04-16T07:45:52.810 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_os_flags": "0", 2024-04-16T07:45:52.810 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_peering_op_priority": "255", 2024-04-16T07:45:52.810 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_delete_cost": "1048576", 2024-04-16T07:45:52.810 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_delete_priority": "5", 2024-04-16T07:45:52.810 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-16T07:45:52.810 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-16T07:45:52.810 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-16T07:45:52.810 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_log_trim_max": "10000", 2024-04-16T07:45:52.810 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_log_trim_min": "100", 2024-04-16T07:45:52.811 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-16T07:45:52.811 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-16T07:45:52.811 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-16T07:45:52.811 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-16T07:45:52.811 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-16T07:45:52.811 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-16T07:45:52.811 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-16T07:45:52.811 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-16T07:45:52.811 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-16T07:45:52.811 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-16T07:45:52.811 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_crimson": "true", 2024-04-16T07:45:52.811 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-16T07:45:52.811 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-16T07:45:52.811 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-16T07:45:52.811 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-16T07:45:52.811 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-16T07:45:52.811 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-16T07:45:52.812 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-16T07:45:52.812 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-16T07:45:52.812 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_flags": "0", 2024-04-16T07:45:52.812 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-16T07:45:52.812 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_min_size": "2", 2024-04-16T07:45:52.812 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-16T07:45:52.812 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_pg_num": "32", 2024-04-16T07:45:52.812 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_pgp_num": "0", 2024-04-16T07:45:52.812 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-16T07:45:52.812 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_read_ratio": "70", 2024-04-16T07:45:52.812 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_size": "3", 2024-04-16T07:45:52.812 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_type": "replicated", 2024-04-16T07:45:52.812 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-16T07:45:52.812 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-16T07:45:52.812 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_push_per_object_cost": "1000", 2024-04-16T07:45:52.812 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-16T07:45:52.812 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recover_clone_overlap": "true", 2024-04-16T07:45:52.813 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-16T07:45:52.813 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_cost": "20971520", 2024-04-16T07:45:52.813 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-16T07:45:52.813 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_max_active": "0", 2024-04-16T07:45:52.813 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-16T07:45:52.813 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-16T07:45:52.813 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-16T07:45:52.813 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-16T07:45:52.813 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_max_single_start": "1", 2024-04-16T07:45:52.813 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_op_priority": "3", 2024-04-16T07:45:52.813 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-16T07:45:52.813 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_priority": "5", 2024-04-16T07:45:52.813 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-16T07:45:52.813 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_sleep": "0.000000", 2024-04-16T07:45:52.813 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-16T07:45:52.813 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-16T07:45:52.814 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-16T07:45:52.814 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_repair_during_recovery": "false", 2024-04-16T07:45:52.814 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_requested_scrub_priority": "120", 2024-04-16T07:45:52.814 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-16T07:45:52.814 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-16T07:45:52.814 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_auto_repair": "false", 2024-04-16T07:45:52.814 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-16T07:45:52.814 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-16T07:45:52.814 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_begin_hour": "0", 2024-04-16T07:45:52.814 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_begin_week_day": "0", 2024-04-16T07:45:52.814 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_chunk_max": "25", 2024-04-16T07:45:52.814 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_chunk_min": "5", 2024-04-16T07:45:52.814 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_cost": "52428800", 2024-04-16T07:45:52.814 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-16T07:45:52.815 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_during_recovery": "false", 2024-04-16T07:45:52.815 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_end_hour": "0", 2024-04-16T07:45:52.815 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_end_week_day": "0", 2024-04-16T07:45:52.815 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_event_cost": "4096", 2024-04-16T07:45:52.815 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-16T07:45:52.815 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-16T07:45:52.815 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_invalid_stats": "true", 2024-04-16T07:45:52.815 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-16T07:45:52.815 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-16T07:45:52.815 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_max_preemptions": "5", 2024-04-16T07:45:52.815 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-16T07:45:52.815 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_priority": "5", 2024-04-16T07:45:52.815 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-16T07:45:52.815 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_sleep": "0.000000", 2024-04-16T07:45:52.815 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-16T07:45:52.815 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-16T07:45:52.815 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-16T07:45:52.816 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-04-16T07:45:52.816 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_skip_data_digest": "false", 2024-04-16T07:45:52.816 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_smart_report_timeout": "5", 2024-04-16T07:45:52.816 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_snap_trim_cost": "1048576", 2024-04-16T07:45:52.816 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_snap_trim_priority": "5", 2024-04-16T07:45:52.816 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-16T07:45:52.816 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-16T07:45:52.816 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-16T07:45:52.816 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-16T07:45:52.816 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-16T07:45:52.816 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-16T07:45:52.816 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-16T07:45:52.816 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_target_transaction_size": "30", 2024-04-16T07:45:52.816 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-16T07:45:52.816 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-16T07:45:52.816 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-16T07:45:52.817 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-16T07:45:52.817 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-16T07:45:52.817 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-16T07:45:52.817 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-16T07:45:52.817 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-16T07:45:52.817 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-16T07:45:52.817 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-16T07:45:52.817 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tracing": "false", 2024-04-16T07:45:52.817 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_use_stale_snap": "false", 2024-04-16T07:45:52.817 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-16T07:45:52.817 INFO:teuthology.orchestra.run.smithi070.stdout: "osdc_blkin_trace_all": "false", 2024-04-16T07:45:52.817 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_kill_at": "0", 2024-04-16T07:45:52.817 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_max_join_drift": "10", 2024-04-16T07:45:52.817 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_min": "500", 2024-04-16T07:45:52.817 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_min_wait": "0.050000", 2024-04-16T07:45:52.817 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_propose_interval": "1.000000", 2024-04-16T07:45:52.817 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_service_trim_max": "500", 2024-04-16T07:45:52.818 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-16T07:45:52.818 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_service_trim_min": "10", 2024-04-16T07:45:52.818 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_stash_full_interval": "25", 2024-04-16T07:45:52.818 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_trim_max": "500", 2024-04-16T07:45:52.818 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_trim_min": "250", 2024-04-16T07:45:52.818 INFO:teuthology.orchestra.run.smithi070.stdout: "perf": "true", 2024-04-16T07:45:52.818 INFO:teuthology.orchestra.run.smithi070.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-16T07:45:52.818 INFO:teuthology.orchestra.run.smithi070.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-16T07:45:52.818 INFO:teuthology.orchestra.run.smithi070.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-16T07:45:52.818 INFO:teuthology.orchestra.run.smithi070.stdout: "public_addr": "-", 2024-04-16T07:45:52.818 INFO:teuthology.orchestra.run.smithi070.stdout: "public_addrv": "", 2024-04-16T07:45:52.818 INFO:teuthology.orchestra.run.smithi070.stdout: "public_bind_addr": "-", 2024-04-16T07:45:52.818 INFO:teuthology.orchestra.run.smithi070.stdout: "public_network": "", 2024-04-16T07:45:52.818 INFO:teuthology.orchestra.run.smithi070.stdout: "public_network_interface": "", 2024-04-16T07:45:52.818 INFO:teuthology.orchestra.run.smithi070.stdout: "qat_compressor_enabled": "false", 2024-04-16T07:45:52.818 INFO:teuthology.orchestra.run.smithi070.stdout: "qat_compressor_session_max_number": "256", 2024-04-16T07:45:52.818 INFO:teuthology.orchestra.run.smithi070.stdout: "rados_mon_op_timeout": "0", 2024-04-16T07:45:52.819 INFO:teuthology.orchestra.run.smithi070.stdout: "rados_osd_op_timeout": "0", 2024-04-16T07:45:52.819 INFO:teuthology.orchestra.run.smithi070.stdout: "rados_tracing": "false", 2024-04-16T07:45:52.819 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_atime_update_interval": "60", 2024-04-16T07:45:52.819 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-16T07:45:52.819 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_balance_parent_reads": "false", 2024-04-16T07:45:52.819 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_balance_snap_reads": "false", 2024-04-16T07:45:52.819 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_blkin_trace_all": "false", 2024-04-16T07:45:52.819 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-16T07:45:52.819 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-16T07:45:52.819 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_cache": "true", 2024-04-16T07:45:52.819 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-16T07:45:52.819 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-16T07:45:52.819 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-16T07:45:52.819 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-16T07:45:52.819 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_cache_policy": "writearound", 2024-04-16T07:45:52.819 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_cache_size": "33554432", 2024-04-16T07:45:52.819 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-16T07:45:52.820 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-16T07:45:52.820 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_clone_copy_on_read": "false", 2024-04-16T07:45:52.820 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_compression_hint": "none", 2024-04-16T07:45:52.820 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_concurrent_management_ops": "10", 2024-04-16T07:45:52.820 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-16T07:45:52.820 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_clone_format": "auto", 2024-04-16T07:45:52.820 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_data_pool": "", 2024-04-16T07:45:52.820 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_features": "61", 2024-04-16T07:45:52.820 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_format": "2", 2024-04-16T07:45:52.820 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_map_options": "", 2024-04-16T07:45:52.820 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_order": "22", 2024-04-16T07:45:52.820 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_pool": "rbd", 2024-04-16T07:45:52.820 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-16T07:45:52.820 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_stripe_count": "0", 2024-04-16T07:45:52.820 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_stripe_unit": "0", 2024-04-16T07:45:52.820 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-16T07:45:52.821 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-16T07:45:52.821 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-16T07:45:52.821 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_enable_alloc_hint": "true", 2024-04-16T07:45:52.821 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-16T07:45:52.821 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_io_scheduler": "simple", 2024-04-16T07:45:52.821 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-16T07:45:52.821 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-16T07:45:52.821 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-16T07:45:52.821 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-16T07:45:52.821 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-16T07:45:52.821 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-16T07:45:52.821 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-16T07:45:52.821 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-16T07:45:52.821 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-16T07:45:52.821 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_order": "24", 2024-04-16T07:45:52.821 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_pool": "", 2024-04-16T07:45:52.821 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_splay_width": "4", 2024-04-16T07:45:52.822 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_localize_parent_reads": "false", 2024-04-16T07:45:52.822 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_localize_snap_reads": "false", 2024-04-16T07:45:52.822 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-16T07:45:52.822 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-16T07:45:52.822 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-16T07:45:52.822 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-16T07:45:52.822 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-16T07:45:52.822 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-16T07:45:52.822 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-16T07:45:52.822 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-16T07:45:52.822 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-16T07:45:52.822 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-16T07:45:52.822 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-16T07:45:52.823 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-16T07:45:52.823 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-16T07:45:52.823 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-16T07:45:52.823 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-16T07:45:52.823 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-16T07:45:52.823 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-16T07:45:52.823 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-16T07:45:52.823 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-16T07:45:52.823 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-16T07:45:52.823 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-16T07:45:52.823 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-16T07:45:52.823 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-16T07:45:52.823 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-16T07:45:52.823 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-16T07:45:52.823 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-16T07:45:52.823 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-16T07:45:52.824 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-16T07:45:52.824 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-16T07:45:52.824 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-16T07:45:52.824 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-16T07:45:52.824 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mtime_update_interval": "60", 2024-04-16T07:45:52.824 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_non_blocking_aio": "true", 2024-04-16T07:45:52.824 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_op_thread_timeout": "60", 2024-04-16T07:45:52.824 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_op_threads": "1", 2024-04-16T07:45:52.824 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_parent_cache_enabled": "false", 2024-04-16T07:45:52.824 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-16T07:45:52.824 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-16T07:45:52.824 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-16T07:45:52.824 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_plugins": "", 2024-04-16T07:45:52.824 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_bps_burst": "0", 2024-04-16T07:45:52.824 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-16T07:45:52.824 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_bps_limit": "0", 2024-04-16T07:45:52.825 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_exclude_ops": "0", 2024-04-16T07:45:52.825 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_iops_burst": "0", 2024-04-16T07:45:52.825 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-16T07:45:52.825 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_iops_limit": "0", 2024-04-16T07:45:52.825 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-16T07:45:52.825 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-16T07:45:52.825 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-16T07:45:52.825 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-16T07:45:52.825 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-16T07:45:52.825 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-16T07:45:52.825 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-16T07:45:52.825 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-16T07:45:52.825 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-16T07:45:52.825 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-16T07:45:52.826 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-16T07:45:52.826 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-16T07:45:52.826 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-16T07:45:52.826 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-16T07:45:52.826 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_read_from_replica_policy": "default", 2024-04-16T07:45:52.826 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-16T07:45:52.826 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-16T07:45:52.826 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-16T07:45:52.826 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-16T07:45:52.826 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_skip_partial_discard": "true", 2024-04-16T07:45:52.826 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-16T07:45:52.826 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_tracing": "false", 2024-04-16T07:45:52.826 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_validate_names": "true", 2024-04-16T07:45:52.826 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_validate_pool": "true", 2024-04-16T07:45:52.826 INFO:teuthology.orchestra.run.smithi070.stdout: "restapi_base_url": "", 2024-04-16T07:45:52.826 INFO:teuthology.orchestra.run.smithi070.stdout: "restapi_log_level": "", 2024-04-16T07:45:52.826 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_acl_grants_max_num": "100", 2024-04-16T07:45:52.827 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_admin_entry": "admin", 2024-04-16T07:45:52.827 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-16T07:45:52.827 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_backend_store": "rados", 2024-04-16T07:45:52.827 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_barbican_url": "", 2024-04-16T07:45:52.827 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_beast_enable_async": "true", 2024-04-16T07:45:52.827 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_bucket_counters_cache": "false", 2024-04-16T07:45:52.827 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-16T07:45:52.827 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-16T07:45:52.827 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-16T07:45:52.827 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-16T07:45:52.827 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-16T07:45:52.827 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-16T07:45:52.827 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-16T07:45:52.827 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-16T07:45:52.827 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_cache_enabled": "true", 2024-04-16T07:45:52.827 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_cache_expiry_interval": "900", 2024-04-16T07:45:52.827 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_cache_lru_size": "10000", 2024-04-16T07:45:52.828 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_config_store": "rados", 2024-04-16T07:45:52.828 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_content_length_compat": "false", 2024-04-16T07:45:52.828 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_copy_obj_progress": "true", 2024-04-16T07:45:52.828 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-16T07:45:52.828 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_cors_rules_max_num": "100", 2024-04-16T07:45:52.828 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_cross_domain_policy": "", 2024-04-16T07:45:52.828 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-16T07:45:52.828 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_kmip_addr": "", 2024-04-16T07:45:52.828 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-16T07:45:52.828 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-16T07:45:52.828 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-16T07:45:52.828 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-16T07:45:52.828 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_kmip_password": "", 2024-04-16T07:45:52.828 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-16T07:45:52.828 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_kmip_username": "", 2024-04-16T07:45:52.828 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_require_ssl": "true", 2024-04-16T07:45:52.828 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-16T07:45:52.829 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-16T07:45:52.829 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-16T07:45:52.829 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-16T07:45:52.829 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-16T07:45:52.829 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-16T07:45:52.829 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-16T07:45:52.829 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-16T07:45:52.829 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-16T07:45:52.829 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-16T07:45:52.829 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-16T07:45:52.829 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-16T07:45:52.829 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-16T07:45:52.829 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-16T07:45:52.829 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-16T07:45:52.829 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_addr": "", 2024-04-16T07:45:52.829 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_auth": "token", 2024-04-16T07:45:52.830 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_namespace": "", 2024-04-16T07:45:52.830 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_prefix": "", 2024-04-16T07:45:52.830 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-16T07:45:52.830 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-16T07:45:52.830 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-16T07:45:52.830 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-16T07:45:52.830 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_token_file": "", 2024-04-16T07:45:52.830 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-16T07:45:52.830 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_curl_buffersize": "524288", 2024-04-16T07:45:52.830 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-16T07:45:52.830 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_curl_low_speed_time": "300", 2024-04-16T07:45:52.830 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-16T07:45:52.830 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-16T07:45:52.830 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-16T07:45:52.830 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-16T07:45:52.830 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-16T07:45:52.831 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-16T07:45:52.831 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-16T07:45:52.831 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-16T07:45:52.831 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-16T07:45:52.831 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-16T07:45:52.831 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-16T07:45:52.831 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-04-16T07:45:52.831 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-04-16T07:45:52.831 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-04-16T07:45:52.831 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d4n_l1_fadvise": "4", 2024-04-16T07:45:52.831 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-04-16T07:45:52.831 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-04-16T07:45:52.831 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d4n_port": "6379", 2024-04-16T07:45:52.831 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-16T07:45:52.831 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_data_log_changes_size": "1000", 2024-04-16T07:45:52.831 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_data_log_num_shards": "128", 2024-04-16T07:45:52.831 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-16T07:45:52.832 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_data_log_window": "30", 2024-04-16T07:45:52.832 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-16T07:45:52.832 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-16T07:45:52.832 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-16T07:45:52.832 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-16T07:45:52.832 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-16T07:45:52.832 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-16T07:45:52.832 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-16T07:45:52.832 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-16T07:45:52.832 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-16T07:45:52.832 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-16T07:45:52.832 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-16T07:45:52.832 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-16T07:45:52.832 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-16T07:45:52.832 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-16T07:45:52.832 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-16T07:45:52.832 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-16T07:45:52.833 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-16T07:45:52.833 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-16T07:45:52.833 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-16T07:45:52.833 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-16T07:45:52.833 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-16T07:45:52.833 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-16T07:45:52.833 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-16T07:45:52.833 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-16T07:45:52.833 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-16T07:45:52.833 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dns_name": "", 2024-04-16T07:45:52.833 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dns_s3website_name": "", 2024-04-16T07:45:52.833 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dynamic_resharding": "true", 2024-04-16T07:45:52.833 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-16T07:45:52.833 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_enable_gc_threads": "true", 2024-04-16T07:45:52.833 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_enable_lc_threads": "true", 2024-04-16T07:45:52.833 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_enable_ops_log": "false", 2024-04-16T07:45:52.833 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_enable_quota_threads": "true", 2024-04-16T07:45:52.834 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_enable_static_website": "false", 2024-04-16T07:45:52.834 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_enable_usage_log": "false", 2024-04-16T07:45:52.834 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_enforce_swift_acls": "true", 2024-04-16T07:45:52.834 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_exit_timeout_secs": "120", 2024-04-16T07:45:52.834 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_expose_bucket": "false", 2024-04-16T07:45:52.834 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_extended_http_attrs": "", 2024-04-16T07:45:52.834 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_filter": "none", 2024-04-16T07:45:52.834 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-16T07:45:52.834 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_frontends": "beast port=7480", 2024-04-16T07:45:52.834 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-16T07:45:52.834 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_gc_max_deferred": "50", 2024-04-16T07:45:52.834 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-16T07:45:52.834 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_gc_max_objs": "32", 2024-04-16T07:45:52.834 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-16T07:45:52.834 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-16T07:45:52.834 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-16T07:45:52.834 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-16T07:45:52.835 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_gc_processor_period": "3600", 2024-04-16T07:45:52.835 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-16T07:45:52.835 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-16T07:45:52.835 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-16T07:45:52.835 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-16T07:45:52.835 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_init_timeout": "300", 2024-04-16T07:45:52.835 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-16T07:45:52.835 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-16T07:45:52.835 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_kafka_connection_idle": "300", 2024-04-16T07:45:52.835 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-16T07:45:52.835 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-16T07:45:52.835 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-16T07:45:52.835 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-16T07:45:52.835 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-16T07:45:52.835 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_admin_domain": "", 2024-04-16T07:45:52.835 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_admin_password": "", 2024-04-16T07:45:52.835 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_admin_password_path": "", 2024-04-16T07:45:52.836 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_admin_project": "", 2024-04-16T07:45:52.836 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_admin_tenant": "", 2024-04-16T07:45:52.836 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_admin_token": "", 2024-04-16T07:45:52.836 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_admin_token_path": "", 2024-04-16T07:45:52.836 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_admin_user": "", 2024-04-16T07:45:52.836 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_api_version": "2", 2024-04-16T07:45:52.836 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_barbican_domain": "", 2024-04-16T07:45:52.836 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_barbican_password": "", 2024-04-16T07:45:52.836 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_barbican_project": "", 2024-04-16T07:45:52.836 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-16T07:45:52.836 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_barbican_user": "", 2024-04-16T07:45:52.836 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-16T07:45:52.836 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-16T07:45:52.836 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-16T07:45:52.836 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-16T07:45:52.836 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-16T07:45:52.837 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_url": "", 2024-04-16T07:45:52.837 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-16T07:45:52.837 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lc_debug_interval": "-1", 2024-04-16T07:45:52.837 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lc_lock_max_time": "90", 2024-04-16T07:45:52.837 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lc_max_objs": "32", 2024-04-16T07:45:52.837 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lc_max_rules": "1000", 2024-04-16T07:45:52.837 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lc_max_worker": "3", 2024-04-16T07:45:52.837 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-16T07:45:52.837 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lc_thread_delay": "0", 2024-04-16T07:45:52.837 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-16T07:45:52.837 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ldap_dnattr": "uid", 2024-04-16T07:45:52.837 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-16T07:45:52.837 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ldap_searchfilter": "", 2024-04-16T07:45:52.837 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-16T07:45:52.838 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ldap_uri": "", 2024-04-16T07:45:52.838 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lfuda_sync_frequency": "60", 2024-04-16T07:45:52.838 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-16T07:45:52.838 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-16T07:45:52.838 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-16T07:45:52.838 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_local_cache_address": "127.0.0.1:6379", 2024-04-16T07:45:52.838 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_log_http_headers": "", 2024-04-16T07:45:52.838 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-16T07:45:52.838 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-16T07:45:52.838 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_log_object_name_utc": "false", 2024-04-16T07:45:52.838 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-16T07:45:52.838 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-16T07:45:52.838 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_attr_name_len": "0", 2024-04-16T07:45:52.838 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_attr_size": "0", 2024-04-16T07:45:52.838 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-16T07:45:52.838 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_chunk_size": "4194304", 2024-04-16T07:45:52.839 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-16T07:45:52.839 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-16T07:45:52.839 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-16T07:45:52.839 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_listing_results": "1000", 2024-04-16T07:45:52.839 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_notify_retries": "10", 2024-04-16T07:45:52.839 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-16T07:45:52.839 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_put_param_size": "1048576", 2024-04-16T07:45:52.839 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_put_size": "5368709120", 2024-04-16T07:45:52.839 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_slo_entries": "1000", 2024-04-16T07:45:52.839 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_md_log_max_shards": "64", 2024-04-16T07:45:52.839 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-16T07:45:52.839 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-16T07:45:52.839 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-16T07:45:52.839 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-16T07:45:52.839 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_mp_lock_max_time": "600", 2024-04-16T07:45:52.839 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-16T07:45:52.839 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-16T07:45:52.840 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-16T07:45:52.840 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-16T07:45:52.840 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-16T07:45:52.840 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-16T07:45:52.840 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-16T07:45:52.840 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-16T07:45:52.840 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_max_gc": "300", 2024-04-16T07:45:52.840 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-16T07:45:52.840 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-16T07:45:52.840 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-16T07:45:52.840 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-16T07:45:52.840 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-16T07:45:52.840 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-16T07:45:52.840 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-16T07:45:52.840 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_num_async_rados_threads": "32", 2024-04-16T07:45:52.840 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_num_control_oids": "8", 2024-04-16T07:45:52.841 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_numa_node": "-1", 2024-04-16T07:45:52.841 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-16T07:45:52.841 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-16T07:45:52.841 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_objexp_chunk_size": "100", 2024-04-16T07:45:52.841 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_objexp_gc_interval": "600", 2024-04-16T07:45:52.841 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-16T07:45:52.841 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-16T07:45:52.841 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-16T07:45:52.841 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_op_thread_timeout": "600", 2024-04-16T07:45:52.841 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_op_tracing": "false", 2024-04-16T07:45:52.841 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_opa_token": "", 2024-04-16T07:45:52.841 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_opa_url": "", 2024-04-16T07:45:52.841 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_opa_verify_ssl": "true", 2024-04-16T07:45:52.841 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-16T07:45:52.841 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-16T07:45:52.841 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ops_log_rados": "false", 2024-04-16T07:45:52.841 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ops_log_socket_path": "", 2024-04-16T07:45:52.842 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-16T07:45:52.842 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-16T07:45:52.842 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-16T07:45:52.842 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_period_push_interval": "2.000000", 2024-04-16T07:45:52.842 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-16T07:45:52.842 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-16T07:45:52.842 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-16T07:45:52.842 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-16T07:45:52.842 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_posix_cache_lanes": "3", 2024-04-16T07:45:52.842 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-16T07:45:52.842 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-16T07:45:52.842 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_posix_cache_partitions": "3", 2024-04-16T07:45:52.842 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-16T07:45:52.842 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_print_continue": "true", 2024-04-16T07:45:52.842 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-16T07:45:52.842 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-16T07:45:52.843 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-16T07:45:52.843 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-16T07:45:52.843 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-16T07:45:52.843 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_rados_tracing": "false", 2024-04-16T07:45:52.843 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_realm": "", 2024-04-16T07:45:52.843 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_realm_id": "", 2024-04-16T07:45:52.843 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-16T07:45:52.843 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_region": "", 2024-04-16T07:45:52.843 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-16T07:45:52.843 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-16T07:45:52.843 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-16T07:45:52.843 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-16T07:45:52.843 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_request_uri": "", 2024-04-16T07:45:52.843 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_reshard_batch_size": "64", 2024-04-16T07:45:52.843 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-16T07:45:52.843 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_reshard_max_aio": "128", 2024-04-16T07:45:52.843 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_reshard_num_logs": "16", 2024-04-16T07:45:52.844 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_reshard_thread_interval": "600", 2024-04-16T07:45:52.844 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_resolve_cname": "false", 2024-04-16T07:45:52.844 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-16T07:45:52.844 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_run_sync_thread": "true", 2024-04-16T07:45:52.844 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-16T07:45:52.844 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-16T07:45:52.844 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-16T07:45:52.844 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-16T07:45:52.844 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-16T07:45:52.844 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-16T07:45:52.844 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-16T07:45:52.844 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-16T07:45:52.844 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-16T07:45:52.844 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_scheduler_type": "throttler", 2024-04-16T07:45:52.844 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_script_uri": "", 2024-04-16T07:45:52.845 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_service_provider_name": "", 2024-04-16T07:45:52.845 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-16T07:45:52.845 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sts_client_id": "", 2024-04-16T07:45:52.845 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sts_client_secret": "", 2024-04-16T07:45:52.845 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sts_entry": "sts", 2024-04-16T07:45:52.845 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sts_key": "sts", 2024-04-16T07:45:52.845 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-16T07:45:52.845 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sts_min_session_duration": "900", 2024-04-16T07:45:52.845 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sts_token_introspection_url": "", 2024-04-16T07:45:52.845 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_account_in_url": "false", 2024-04-16T07:45:52.845 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_auth_entry": "auth", 2024-04-16T07:45:52.845 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_auth_url": "", 2024-04-16T07:45:52.845 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_custom_header": "", 2024-04-16T07:45:52.845 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-16T07:45:52.845 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_need_stats": "true", 2024-04-16T07:45:52.845 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_tenant_name": "", 2024-04-16T07:45:52.846 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_token_expiration": "86400", 2024-04-16T07:45:52.846 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_url": "", 2024-04-16T07:45:52.846 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_url_prefix": "swift", 2024-04-16T07:45:52.846 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-16T07:45:52.846 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-16T07:45:52.846 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-16T07:45:52.846 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_lease_period": "120", 2024-04-16T07:45:52.846 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-16T07:45:52.846 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-16T07:45:52.846 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-16T07:45:52.846 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-16T07:45:52.846 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-16T07:45:52.846 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-16T07:45:52.846 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-16T07:45:52.846 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-16T07:45:52.847 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-16T07:45:52.847 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_thread_pool_size": "512", 2024-04-16T07:45:52.847 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-16T07:45:52.847 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-16T07:45:52.847 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-16T07:45:52.847 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-16T07:45:52.847 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_torrent_comment": "", 2024-04-16T07:45:52.847 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_torrent_createby": "", 2024-04-16T07:45:52.847 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_torrent_encoding": "", 2024-04-16T07:45:52.847 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_torrent_flag": "false", 2024-04-16T07:45:52.847 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-16T07:45:52.847 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_torrent_origin": "", 2024-04-16T07:45:52.847 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-16T07:45:52.847 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_torrent_tracker": "", 2024-04-16T07:45:52.847 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_trust_forwarded_https": "false", 2024-04-16T07:45:52.847 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-16T07:45:52.847 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-16T07:45:52.848 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_usage_max_shards": "32", 2024-04-16T07:45:52.848 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_usage_max_user_shards": "1", 2024-04-16T07:45:52.848 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_use_opa_authz": "false", 2024-04-16T07:45:52.848 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_counters_cache": "false", 2024-04-16T07:45:52.848 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-16T07:45:52.848 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-16T07:45:52.848 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-16T07:45:52.848 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_max_buckets": "1000", 2024-04-16T07:45:52.848 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_policies_max_num": "100", 2024-04-16T07:45:52.848 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-16T07:45:52.848 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-16T07:45:52.848 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-16T07:45:52.848 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-16T07:45:52.848 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_unique_email": "true", 2024-04-16T07:45:52.848 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_verify_ssl": "true", 2024-04-16T07:45:52.848 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-16T07:45:52.848 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_zone": "", 2024-04-16T07:45:52.849 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_zone_id": "", 2024-04-16T07:45:52.849 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-16T07:45:52.849 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_zonegroup": "", 2024-04-16T07:45:52.849 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_zonegroup_id": "", 2024-04-16T07:45:52.849 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-16T07:45:52.849 INFO:teuthology.orchestra.run.smithi070.stdout: "rgwlc_auto_session_clear": "true", 2024-04-16T07:45:52.849 INFO:teuthology.orchestra.run.smithi070.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-16T07:45:52.849 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_block_size": "4096", 2024-04-16T07:45:52.849 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-16T07:45:52.849 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-16T07:45:52.849 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-16T07:45:52.849 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-16T07:45:52.849 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-16T07:45:52.849 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_cache_size": "536870912", 2024-04-16T07:45:52.849 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-16T07:45:52.849 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-16T07:45:52.850 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-16T07:45:52.850 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-16T07:45:52.850 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-16T07:45:52.850 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-16T07:45:52.850 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-16T07:45:52.850 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-16T07:45:52.850 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_index_type": "binary_search", 2024-04-16T07:45:52.850 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-16T07:45:52.850 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-16T07:45:52.850 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_partition_filters": "false", 2024-04-16T07:45:52.850 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_perf": "false", 2024-04-16T07:45:52.850 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-16T07:45:52.850 INFO:teuthology.orchestra.run.smithi070.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-16T07:45:52.850 INFO:teuthology.orchestra.run.smithi070.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-16T07:45:52.850 INFO:teuthology.orchestra.run.smithi070.stdout: "run_dir": "/var/run/ceph", 2024-04-16T07:45:52.850 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_block_create": "true", 2024-04-16T07:45:52.850 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_cache_lru_size": "67108864", 2024-04-16T07:45:52.851 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-16T07:45:52.851 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-16T07:45:52.851 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_default_max_object_size": "16777216", 2024-04-16T07:45:52.851 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-16T07:45:52.851 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_device_size": "53687091200", 2024-04-16T07:45:52.851 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_journal_batch_capacity": "16", 2024-04-16T07:45:52.851 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-16T07:45:52.851 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-16T07:45:52.851 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-16T07:45:52.851 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_main_device_type": "SSD", 2024-04-16T07:45:52.851 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-16T07:45:52.851 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-16T07:45:52.851 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-16T07:45:52.851 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-16T07:45:52.851 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-16T07:45:52.851 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_segment_size": "67108864", 2024-04-16T07:45:52.851 INFO:teuthology.orchestra.run.smithi070.stdout: "set_keepcaps": "false", 2024-04-16T07:45:52.852 INFO:teuthology.orchestra.run.smithi070.stdout: "setgroup": "", 2024-04-16T07:45:52.852 INFO:teuthology.orchestra.run.smithi070.stdout: "setuser": "", 2024-04-16T07:45:52.852 INFO:teuthology.orchestra.run.smithi070.stdout: "setuser_match_path": "", 2024-04-16T07:45:52.852 INFO:teuthology.orchestra.run.smithi070.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-16T07:45:52.852 INFO:teuthology.orchestra.run.smithi070.stdout: "thp": "false", 2024-04-16T07:45:52.852 INFO:teuthology.orchestra.run.smithi070.stdout: "threadpool_default_timeout": "60", 2024-04-16T07:45:52.852 INFO:teuthology.orchestra.run.smithi070.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-16T07:45:52.852 INFO:teuthology.orchestra.run.smithi070.stdout: "throttler_perf_counter": "true" 2024-04-16T07:45:52.852 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-16T07:45:52.854 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-16T07:45:52.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:52.963+0000 7f32a56ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:52.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:52.969+0000 7f32a56ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:52.982 INFO:teuthology.orchestra.run.smithi070.stdout:{} 2024-04-16T07:45:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-16T07:45:52.993 DEBUG:teuthology.orchestra.run.smithi172:> type ceph-objectstore-tool 2024-04-16T07:45:53.012 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-16T07:45:53.012 DEBUG:teuthology.orchestra.run.smithi070:> type ceph-objectstore-tool 2024-04-16T07:45:53.027 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-16T07:45:53.027 INFO:teuthology.run_tasks:Running task rados... 2024-04-16T07:45:53.037 INFO:tasks.rados:Beginning rados... 2024-04-16T07:45:53.037 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 500, 'op_weights': {'write': 100}, 'ops': 4000, 'write_fadvise_dontneed': True} 2024-04-16T07:45:53.037 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-matanb-crimson-replicated-april-15'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-matanb-crimson-replicated-april-15', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}} 2024-04-16T07:45:53.037 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 500, 'op_weights': {'write': 100}, 'ops': 4000, 'write_fadvise_dontneed': True} 2024-04-16T07:45:53.037 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-04-16T07:45:53.045 INFO:tasks.rados:joining rados 2024-04-16T07:45:53.045 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-16T07:45:53.045 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-16T07:45:53.045 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:45:53.045 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:45:53.046 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-16T07:45:53.046 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:45:53.046 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2024-04-16T07:45:53.046 INFO:tasks.rados:starting run 0 out of 1 2024-04-16T07:45:53.046 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-16T07:45:53.046 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-04-16T07:45:53.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:53.182+0000 7f00f962d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:53.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:53.182+0000 7f45a9a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:53.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:53.188+0000 7f00f962d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:53.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:53.190+0000 7f45a9a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:53.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:53.182+0000 7fec8ac64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:53.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:53.203+0000 7fec8ac64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:53.386 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:45:53.386 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-16T07:45:53.402 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-16T07:45:51.619113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107912,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379448,"statfs":{"total":386547056640,"available":386436554752,"internally_reserved":0,"allocated":593920,"data_stored":1870024,"data_compressed":89727,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.965235"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:49.652256+0000","last_change":"2024-04-16T07:45:48.769721+0000","last_active":"2024-04-16T07:45:49.652256+0000","last_peered":"2024-04-16T07:45:49.652256+0000","last_clean":"2024-04-16T07:45:49.652256+0000","last_became_active":"2024-04-16T07:45:48.764515+0000","last_became_peered":"2024-04-16T07:45:48.764515+0000","last_unstale":"2024-04-16T07:45:49.652256+0000","last_undegraded":"2024-04-16T07:45:49.652256+0000","last_fullsized":"2024-04-16T07:45:49.652256+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.765070+0000","last_change":"2024-04-16T07:45:48.766237+0000","last_active":"2024-04-16T07:45:48.765070+0000","last_peered":"2024-04-16T07:45:48.765070+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.761479+0000","last_became_peered":"2024-04-16T07:45:48.761479+0000","last_unstale":"2024-04-16T07:45:48.765070+0000","last_undegraded":"2024-04-16T07:45:48.765070+0000","last_fullsized":"2024-04-16T07:45:48.765070+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.765103+0000","last_change":"2024-04-16T07:45:48.766846+0000","last_active":"2024-04-16T07:45:48.765103+0000","last_peered":"2024-04-16T07:45:48.765103+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.761593+0000","last_became_peered":"2024-04-16T07:45:48.761593+0000","last_unstale":"2024-04-16T07:45:48.765103+0000","last_undegraded":"2024-04-16T07:45:48.765103+0000","last_fullsized":"2024-04-16T07:45:48.765103+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.717426+0000","last_change":"2024-04-16T07:45:48.718558+0000","last_active":"2024-04-16T07:45:48.717426+0000","last_peered":"2024-04-16T07:45:48.717426+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.713866+0000","last_became_peered":"2024-04-16T07:45:48.713866+0000","last_unstale":"2024-04-16T07:45:48.717426+0000","last_undegraded":"2024-04-16T07:45:48.717426+0000","last_fullsized":"2024-04-16T07:45:48.717426+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.773933+0000","last_change":"2024-04-16T07:45:48.774974+0000","last_active":"2024-04-16T07:45:48.773933+0000","last_peered":"2024-04-16T07:45:48.773933+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.768713+0000","last_became_peered":"2024-04-16T07:45:48.768713+0000","last_unstale":"2024-04-16T07:45:48.773933+0000","last_undegraded":"2024-04-16T07:45:48.773933+0000","last_fullsized":"2024-04-16T07:45:48.773933+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:49.648519+0000","last_change":"2024-04-16T07:45:48.724954+0000","last_active":"2024-04-16T07:45:49.648519+0000","last_peered":"2024-04-16T07:45:49.648519+0000","last_clean":"2024-04-16T07:45:49.648519+0000","last_became_active":"2024-04-16T07:45:48.719985+0000","last_became_peered":"2024-04-16T07:45:48.719985+0000","last_unstale":"2024-04-16T07:45:49.648519+0000","last_undegraded":"2024-04-16T07:45:49.648519+0000","last_fullsized":"2024-04-16T07:45:49.648519+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:48.811632+0000","last_change":"2024-04-16T07:45:48.812462+0000","last_active":"2024-04-16T07:45:48.811632+0000","last_peered":"2024-04-16T07:45:48.811632+0000","last_clean":"2024-04-16T07:45:47.630254+0000","last_became_active":"2024-04-16T07:45:48.804250+0000","last_became_peered":"2024-04-16T07:45:48.804250+0000","last_unstale":"2024-04-16T07:45:48.811632+0000","last_undegraded":"2024-04-16T07:45:48.811632+0000","last_fullsized":"2024-04-16T07:45:48.811632+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:49.652389+0000","last_change":"2024-04-16T07:45:48.796564+0000","last_active":"2024-04-16T07:45:49.652389+0000","last_peered":"2024-04-16T07:45:49.652389+0000","last_clean":"2024-04-16T07:45:49.652389+0000","last_became_active":"2024-04-16T07:45:48.789023+0000","last_became_peered":"2024-04-16T07:45:48.789023+0000","last_unstale":"2024-04-16T07:45:49.652389+0000","last_undegraded":"2024-04-16T07:45:49.652389+0000","last_fullsized":"2024-04-16T07:45:49.652389+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:49.351146+0000","last_change":"2024-04-16T07:45:47.694188+0000","last_active":"2024-04-16T07:45:49.351146+0000","last_peered":"2024-04-16T07:45:49.351146+0000","last_clean":"2024-04-16T07:45:49.351146+0000","last_became_active":"2024-04-16T07:45:47.687698+0000","last_became_peered":"2024-04-16T07:45:47.687698+0000","last_unstale":"2024-04-16T07:45:49.351146+0000","last_undegraded":"2024-04-16T07:45:49.351146+0000","last_fullsized":"2024-04-16T07:45:49.351146+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_clean_scrub_stamp":"2024-04-16T07:45:46.626705+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:45:53.656 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'unique_pool_0' created 2024-04-16T07:45:53.676 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-16T07:45:53.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:53.808+0000 7f156279d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:53.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:53.821+0000 7f156279d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:54.659 INFO:teuthology.orchestra.run.smithi070.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-16T07:45:55.298 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:45:55.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:55.411+0000 7f2555a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:55.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:55.416+0000 7f2555a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:55.679 DEBUG:teuthology.orchestra.run.smithi070:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0 2024-04-16T07:45:55.714 INFO:tasks.rados.rados.0.smithi070.stdout:adding op weight read -> 100 2024-04-16T07:45:55.715 INFO:tasks.rados.rados.0.smithi070.stdout:adding op weight write -> 50 2024-04-16T07:45:55.715 INFO:tasks.rados.rados.0.smithi070.stdout:adding op weight delete -> 10 2024-04-16T07:45:55.715 INFO:tasks.rados.rados.0.smithi070.stdout:adding op weight write_excl -> 50 2024-04-16T07:45:55.715 INFO:tasks.rados.rados.0.smithi070.stdout:ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev) 2024-04-16T07:45:55.715 INFO:tasks.rados.rados.0.smithi070.stdout:Configuration: 2024-04-16T07:45:55.715 INFO:tasks.rados.rados.0.smithi070.stdout: Number of operations: 4000 2024-04-16T07:45:55.715 INFO:tasks.rados.rados.0.smithi070.stdout: Number of objects: 500 2024-04-16T07:45:55.715 INFO:tasks.rados.rados.0.smithi070.stdout: Max in flight operations: 16 2024-04-16T07:45:55.715 INFO:tasks.rados.rados.0.smithi070.stdout: Object size (in bytes): 4000000 2024-04-16T07:45:55.715 INFO:tasks.rados.rados.0.smithi070.stdout: Write stride min: 400000 2024-04-16T07:45:55.715 INFO:tasks.rados.rados.0.smithi070.stdout: Write stride max: 800000 2024-04-16T07:45:56.663 INFO:tasks.rados.rados.0.smithi070.stdout:1: write initial oid 1 2024-04-16T07:45:56.663 INFO:tasks.rados.rados.0.smithi070.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-04-16T07:45:56.670 INFO:tasks.rados.rados.0.smithi070.stdout:1: writing smithi07031500-1 from 416807 to 892056 tid 1 2024-04-16T07:45:56.680 INFO:tasks.rados.rados.0.smithi070.stdout:1: writing smithi07031500-1 from 1542129 to 2016807 tid 2 2024-04-16T07:45:56.680 INFO:tasks.rados.rados.0.smithi070.stdout:2: write initial oid 2 2024-04-16T07:45:56.680 INFO:tasks.rados.rados.0.smithi070.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-04-16T07:45:56.687 INFO:tasks.rados.rados.0.smithi070.stdout:2: writing smithi07031500-2 from 416807 to 892056 tid 1 2024-04-16T07:45:56.696 INFO:tasks.rados.rados.0.smithi070.stdout:2: writing smithi07031500-2 from 1542129 to 2016807 tid 2 2024-04-16T07:45:56.696 INFO:tasks.rados.rados.0.smithi070.stdout:3: write initial oid 3 2024-04-16T07:45:56.696 INFO:tasks.rados.rados.0.smithi070.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-04-16T07:45:56.704 INFO:tasks.rados.rados.0.smithi070.stdout:3: writing smithi07031500-3 from 433614 to 984112 tid 1 2024-04-16T07:45:56.711 INFO:tasks.rados.rados.0.smithi070.stdout:3: writing smithi07031500-3 from 1600611 to 2033614 tid 2 2024-04-16T07:45:56.711 INFO:tasks.rados.rados.0.smithi070.stdout:4: write initial oid 4 2024-04-16T07:45:56.711 INFO:tasks.rados.rados.0.smithi070.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-04-16T07:45:56.718 INFO:tasks.rados.rados.0.smithi070.stdout:4: writing smithi07031500-4 from 450421 to 1076168 tid 1 2024-04-16T07:45:56.725 INFO:tasks.rados.rados.0.smithi070.stdout:4: writing smithi07031500-4 from 1659093 to 2050421 tid 2 2024-04-16T07:45:56.725 INFO:tasks.rados.rados.0.smithi070.stdout:5: write initial oid 5 2024-04-16T07:45:56.725 INFO:tasks.rados.rados.0.smithi070.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-04-16T07:45:56.732 INFO:tasks.rados.rados.0.smithi070.stdout:5: writing smithi07031500-5 from 467228 to 1168224 tid 1 2024-04-16T07:45:56.740 INFO:tasks.rados.rados.0.smithi070.stdout:5: writing smithi07031500-5 from 1717575 to 2067228 tid 2 2024-04-16T07:45:56.740 INFO:tasks.rados.rados.0.smithi070.stdout:6: write initial oid 6 2024-04-16T07:45:56.740 INFO:tasks.rados.rados.0.smithi070.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-04-16T07:45:56.749 INFO:tasks.rados.rados.0.smithi070.stdout:6: writing smithi07031500-6 from 484035 to 1260280 tid 1 2024-04-16T07:45:56.754 INFO:tasks.rados.rados.0.smithi070.stdout:6: writing smithi07031500-6 from 2059704 to 2084035 tid 2 2024-04-16T07:45:56.754 INFO:tasks.rados.rados.0.smithi070.stdout:7: write initial oid 7 2024-04-16T07:45:56.754 INFO:tasks.rados.rados.0.smithi070.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-04-16T07:45:56.761 INFO:tasks.rados.rados.0.smithi070.stdout:7: writing smithi07031500-7 from 500842 to 952336 tid 1 2024-04-16T07:45:56.768 INFO:tasks.rados.rados.0.smithi070.stdout:7: writing smithi07031500-7 from 1718186 to 2100842 tid 2 2024-04-16T07:45:56.768 INFO:tasks.rados.rados.0.smithi070.stdout:8: write initial oid 8 2024-04-16T07:45:56.768 INFO:tasks.rados.rados.0.smithi070.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-04-16T07:45:56.775 INFO:tasks.rados.rados.0.smithi070.stdout:8: writing smithi07031500-8 from 517649 to 1044392 tid 1 2024-04-16T07:45:56.782 INFO:tasks.rados.rados.0.smithi070.stdout:8: writing smithi07031500-8 from 1776668 to 2117649 tid 2 2024-04-16T07:45:56.783 INFO:tasks.rados.rados.0.smithi070.stdout:9: write initial oid 9 2024-04-16T07:45:56.783 INFO:tasks.rados.rados.0.smithi070.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-04-16T07:45:56.791 INFO:tasks.rados.rados.0.smithi070.stdout:9: writing smithi07031500-9 from 534456 to 1252801 tid 1 2024-04-16T07:45:56.796 INFO:tasks.rados.rados.0.smithi070.stdout:9: writing smithi07031500-9 from 1951503 to 2134456 tid 2 2024-04-16T07:45:56.796 INFO:tasks.rados.rados.0.smithi070.stdout:10: write initial oid 10 2024-04-16T07:45:56.796 INFO:tasks.rados.rados.0.smithi070.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-04-16T07:45:56.807 INFO:tasks.rados.rados.0.smithi070.stdout:10: writing smithi07031500-10 from 551263 to 1344857 tid 1 2024-04-16T07:45:56.812 INFO:tasks.rados.rados.0.smithi070.stdout:10: writing smithi07031500-10 from 1893632 to 2151263 tid 2 2024-04-16T07:45:56.812 INFO:tasks.rados.rados.0.smithi070.stdout:11: write initial oid 11 2024-04-16T07:45:56.812 INFO:tasks.rados.rados.0.smithi070.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-04-16T07:45:56.818 INFO:tasks.rados.rados.0.smithi070.stdout:11: writing smithi07031500-11 from 568070 to 1036913 tid 1 2024-04-16T07:45:56.826 INFO:tasks.rados.rados.0.smithi070.stdout:11: writing smithi07031500-11 from 1552114 to 2168070 tid 2 2024-04-16T07:45:56.826 INFO:tasks.rados.rados.0.smithi070.stdout:12: write initial oid 12 2024-04-16T07:45:56.826 INFO:tasks.rados.rados.0.smithi070.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-04-16T07:45:56.833 INFO:tasks.rados.rados.0.smithi070.stdout:12: writing smithi07031500-12 from 584877 to 1128969 tid 1 2024-04-16T07:45:56.842 INFO:tasks.rados.rados.0.smithi070.stdout:12: writing smithi07031500-12 from 1610596 to 2172599 tid 2 2024-04-16T07:45:56.842 INFO:tasks.rados.rados.0.smithi070.stdout:13: write initial oid 13 2024-04-16T07:45:56.842 INFO:tasks.rados.rados.0.smithi070.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-04-16T07:45:56.850 INFO:tasks.rados.rados.0.smithi070.stdout:13: writing smithi07031500-13 from 601684 to 1221025 tid 1 2024-04-16T07:45:56.856 INFO:tasks.rados.rados.0.smithi070.stdout:13: writing smithi07031500-13 from 1669078 to 2201684 tid 2 2024-04-16T07:45:56.856 INFO:tasks.rados.rados.0.smithi070.stdout:14: write initial oid 14 2024-04-16T07:45:56.856 INFO:tasks.rados.rados.0.smithi070.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-04-16T07:45:56.864 INFO:tasks.rados.rados.0.smithi070.stdout:14: writing smithi07031500-14 from 618491 to 1313081 tid 1 2024-04-16T07:45:56.870 INFO:tasks.rados.rados.0.smithi070.stdout:14: writing smithi07031500-14 from 2011207 to 2218491 tid 2 2024-04-16T07:45:56.870 INFO:tasks.rados.rados.0.smithi070.stdout:15: write initial oid 15 2024-04-16T07:45:56.870 INFO:tasks.rados.rados.0.smithi070.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-04-16T07:45:56.879 INFO:tasks.rados.rados.0.smithi070.stdout:15: writing smithi07031500-15 from 635298 to 1405137 tid 1 2024-04-16T07:45:56.884 INFO:tasks.rados.rados.0.smithi070.stdout:15: writing smithi07031500-15 from 2069689 to 2235298 tid 2 2024-04-16T07:45:56.885 INFO:tasks.rados.rados.0.smithi070.stdout:16: write initial oid 16 2024-04-16T07:45:56.885 INFO:tasks.rados.rados.0.smithi070.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-04-16T07:45:56.891 INFO:tasks.rados.rados.0.smithi070.stdout:16: writing smithi07031500-16 from 652105 to 1097193 tid 1 2024-04-16T07:45:56.900 INFO:tasks.rados.rados.0.smithi070.stdout:16: writing smithi07031500-16 from 1728171 to 2181159 tid 2 2024-04-16T07:45:56.900 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:56.900 INFO:tasks.rados.rados.0.smithi070.stdout:1: finishing write tid 1 to smithi07031500-1 2024-04-16T07:45:56.900 INFO:tasks.rados.rados.0.smithi070.stdout:3: finishing write tid 1 to smithi07031500-3 2024-04-16T07:45:56.900 INFO:tasks.rados.rados.0.smithi070.stdout:8: finishing write tid 1 to smithi07031500-8 2024-04-16T07:45:56.900 INFO:tasks.rados.rados.0.smithi070.stdout:1: finishing write tid 2 to smithi07031500-1 2024-04-16T07:45:56.900 INFO:tasks.rados.rados.0.smithi070.stdout:3: finishing write tid 2 to smithi07031500-3 2024-04-16T07:45:56.900 INFO:tasks.rados.rados.0.smithi070.stdout:6: finishing write tid 1 to smithi07031500-6 2024-04-16T07:45:56.900 INFO:tasks.rados.rados.0.smithi070.stdout:5: finishing write tid 1 to smithi07031500-5 2024-04-16T07:45:56.901 INFO:tasks.rados.rados.0.smithi070.stdout:8: finishing write tid 2 to smithi07031500-8 2024-04-16T07:45:56.901 INFO:tasks.rados.rados.0.smithi070.stdout:1: finishing write tid 3 to smithi07031500-1 2024-04-16T07:45:56.901 INFO:tasks.rados.rados.0.smithi070.stdout:1: finishing write tid 4 to smithi07031500-1 2024-04-16T07:45:56.901 INFO:tasks.rados.rados.0.smithi070.stdout:1: oid 1 updating version 0 to 1 2024-04-16T07:45:56.901 INFO:tasks.rados.rados.0.smithi070.stdout:1: oid 1 updating version 1 to 2 2024-04-16T07:45:56.901 INFO:tasks.rados.rados.0.smithi070.stdout:1: oid 1 updating version 2 to 3 2024-04-16T07:45:56.901 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-16T07:45:56.901 INFO:tasks.rados.rados.0.smithi070.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-04-16T07:45:56.901 INFO:tasks.rados.rados.0.smithi070.stdout:10: finishing write tid 1 to smithi07031500-10 2024-04-16T07:45:56.901 INFO:tasks.rados.rados.0.smithi070.stdout:8: finishing write tid 3 to smithi07031500-8 2024-04-16T07:45:56.901 INFO:tasks.rados.rados.0.smithi070.stdout:1: done (15 left) 2024-04-16T07:45:56.902 INFO:tasks.rados.rados.0.smithi070.stdout:17: write initial oid 17 2024-04-16T07:45:56.902 INFO:tasks.rados.rados.0.smithi070.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-04-16T07:45:56.909 INFO:tasks.rados.rados.0.smithi070.stdout:17: writing smithi07031500-17 from 668912 to 1305602 tid 1 2024-04-16T07:45:56.917 INFO:tasks.rados.rados.0.smithi070.stdout:17: writing smithi07031500-17 from 1903006 to 2268912 tid 2 2024-04-16T07:45:56.917 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:56.918 INFO:tasks.rados.rados.0.smithi070.stdout:6: finishing write tid 2 to smithi07031500-6 2024-04-16T07:45:56.918 INFO:tasks.rados.rados.0.smithi070.stdout:3: finishing write tid 3 to smithi07031500-3 2024-04-16T07:45:56.918 INFO:tasks.rados.rados.0.smithi070.stdout:8: finishing write tid 4 to smithi07031500-8 2024-04-16T07:45:56.918 INFO:tasks.rados.rados.0.smithi070.stdout:8: oid 8 updating version 0 to 3 2024-04-16T07:45:56.918 INFO:tasks.rados.rados.0.smithi070.stdout:8: oid 8 version 3 is already newer than 3 2024-04-16T07:45:56.918 INFO:tasks.rados.rados.0.smithi070.stdout:8: oid 8 version 3 is already newer than 1 2024-04-16T07:45:56.918 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-16T07:45:56.918 INFO:tasks.rados.rados.0.smithi070.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-04-16T07:45:56.918 INFO:tasks.rados.rados.0.smithi070.stdout:3: finishing write tid 4 to smithi07031500-3 2024-04-16T07:45:56.918 INFO:tasks.rados.rados.0.smithi070.stdout:3: oid 3 updating version 0 to 1 2024-04-16T07:45:56.918 INFO:tasks.rados.rados.0.smithi070.stdout:3: oid 3 updating version 1 to 2 2024-04-16T07:45:56.918 INFO:tasks.rados.rados.0.smithi070.stdout:3: oid 3 updating version 2 to 3 2024-04-16T07:45:56.918 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-16T07:45:56.918 INFO:tasks.rados.rados.0.smithi070.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-04-16T07:45:56.918 INFO:tasks.rados.rados.0.smithi070.stdout:13: finishing write tid 1 to smithi07031500-13 2024-04-16T07:45:56.919 INFO:tasks.rados.rados.0.smithi070.stdout:3: done (15 left) 2024-04-16T07:45:56.919 INFO:tasks.rados.rados.0.smithi070.stdout:8: done (14 left) 2024-04-16T07:45:56.919 INFO:tasks.rados.rados.0.smithi070.stdout:18: write initial oid 18 2024-04-16T07:45:56.919 INFO:tasks.rados.rados.0.smithi070.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-04-16T07:45:56.927 INFO:tasks.rados.rados.0.smithi070.stdout:18: writing smithi07031500-18 from 685719 to 1397658 tid 1 2024-04-16T07:45:56.932 INFO:tasks.rados.rados.0.smithi070.stdout:18: writing smithi07031500-18 from 1845135 to 2285719 tid 2 2024-04-16T07:45:56.933 INFO:tasks.rados.rados.0.smithi070.stdout:19: write initial oid 19 2024-04-16T07:45:56.933 INFO:tasks.rados.rados.0.smithi070.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-04-16T07:45:56.943 INFO:tasks.rados.rados.0.smithi070.stdout:19: writing smithi07031500-19 from 702526 to 1489714 tid 1 2024-04-16T07:45:56.948 INFO:tasks.rados.rados.0.smithi070.stdout:19: writing smithi07031500-19 from 1903617 to 2302526 tid 2 2024-04-16T07:45:56.948 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:56.948 INFO:tasks.rados.rados.0.smithi070.stdout:5: finishing write tid 2 to smithi07031500-5 2024-04-16T07:45:56.948 INFO:tasks.rados.rados.0.smithi070.stdout:2: finishing write tid 1 to smithi07031500-2 2024-04-16T07:45:56.948 INFO:tasks.rados.rados.0.smithi070.stdout:15: finishing write tid 1 to smithi07031500-15 2024-04-16T07:45:56.960 INFO:tasks.rados.rados.0.smithi070.stdout:16: finishing write tid 1 to smithi07031500-16 2024-04-16T07:45:56.964 INFO:tasks.rados.rados.0.smithi070.stdout:12: finishing write tid 1 to smithi07031500-12 2024-04-16T07:45:56.968 INFO:tasks.rados.rados.0.smithi070.stdout:13: finishing write tid 2 to smithi07031500-13 2024-04-16T07:45:56.968 INFO:tasks.rados.rados.0.smithi070.stdout:17: finishing write tid 1 to smithi07031500-17 2024-04-16T07:45:56.983 INFO:tasks.rados.rados.0.smithi070.stdout:2: finishing write tid 2 to smithi07031500-2 2024-04-16T07:45:56.988 INFO:tasks.rados.rados.0.smithi070.stdout:5: finishing write tid 3 to smithi07031500-5 2024-04-16T07:45:56.991 INFO:tasks.rados.rados.0.smithi070.stdout:2: finishing write tid 3 to smithi07031500-2 2024-04-16T07:45:56.993 INFO:tasks.rados.rados.0.smithi070.stdout:10: finishing write tid 2 to smithi07031500-10 2024-04-16T07:45:56.993 INFO:tasks.rados.rados.0.smithi070.stdout:5: finishing write tid 4 to smithi07031500-5 2024-04-16T07:45:56.994 INFO:tasks.rados.rados.0.smithi070.stdout:5: oid 5 updating version 0 to 1 2024-04-16T07:45:56.994 INFO:tasks.rados.rados.0.smithi070.stdout:5: oid 5 updating version 1 to 2 2024-04-16T07:45:56.994 INFO:tasks.rados.rados.0.smithi070.stdout:5: oid 5 updating version 2 to 3 2024-04-16T07:45:56.994 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-16T07:45:56.994 INFO:tasks.rados.rados.0.smithi070.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-04-16T07:45:56.994 INFO:tasks.rados.rados.0.smithi070.stdout:5: done (15 left) 2024-04-16T07:45:56.994 INFO:tasks.rados.rados.0.smithi070.stdout:20: write initial oid 20 2024-04-16T07:45:56.994 INFO:tasks.rados.rados.0.smithi070.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-04-16T07:45:57.002 INFO:tasks.rados.rados.0.smithi070.stdout:20: writing smithi07031500-20 from 719333 to 1181770 tid 1 2024-04-16T07:45:57.010 INFO:tasks.rados.rados.0.smithi070.stdout:20: writing smithi07031500-20 from 1962099 to 2319333 tid 2 2024-04-16T07:45:57.011 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.011 INFO:tasks.rados.rados.0.smithi070.stdout:15: finishing write tid 2 to smithi07031500-15 2024-04-16T07:45:57.011 INFO:tasks.rados.rados.0.smithi070.stdout:6: finishing write tid 3 to smithi07031500-6 2024-04-16T07:45:57.011 INFO:tasks.rados.rados.0.smithi070.stdout:18: finishing write tid 1 to smithi07031500-18 2024-04-16T07:45:57.011 INFO:tasks.rados.rados.0.smithi070.stdout:2: finishing write tid 4 to smithi07031500-2 2024-04-16T07:45:57.011 INFO:tasks.rados.rados.0.smithi070.stdout:2: oid 2 updating version 0 to 4 2024-04-16T07:45:57.011 INFO:tasks.rados.rados.0.smithi070.stdout:2: oid 2 updating version 4 to 6 2024-04-16T07:45:57.011 INFO:tasks.rados.rados.0.smithi070.stdout:2: oid 2 version 6 is already newer than 6 2024-04-16T07:45:57.011 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 2 v 6 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-16T07:45:57.011 INFO:tasks.rados.rados.0.smithi070.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-04-16T07:45:57.011 INFO:tasks.rados.rados.0.smithi070.stdout:17: finishing write tid 2 to smithi07031500-17 2024-04-16T07:45:57.011 INFO:tasks.rados.rados.0.smithi070.stdout:2: done (15 left) 2024-04-16T07:45:57.011 INFO:tasks.rados.rados.0.smithi070.stdout:21: write initial oid 21 2024-04-16T07:45:57.011 INFO:tasks.rados.rados.0.smithi070.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-04-16T07:45:57.020 INFO:tasks.rados.rados.0.smithi070.stdout:21: writing smithi07031500-21 from 736140 to 1273826 tid 1 2024-04-16T07:45:57.028 INFO:tasks.rados.rados.0.smithi070.stdout:21: writing smithi07031500-21 from 2020581 to 2336140 tid 2 2024-04-16T07:45:57.028 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.028 INFO:tasks.rados.rados.0.smithi070.stdout:6: finishing write tid 4 to smithi07031500-6 2024-04-16T07:45:57.028 INFO:tasks.rados.rados.0.smithi070.stdout:6: oid 6 updating version 0 to 2 2024-04-16T07:45:57.028 INFO:tasks.rados.rados.0.smithi070.stdout:6: oid 6 version 2 is already newer than 1 2024-04-16T07:45:57.029 INFO:tasks.rados.rados.0.smithi070.stdout:6: oid 6 updating version 2 to 3 2024-04-16T07:45:57.029 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-16T07:45:57.029 INFO:tasks.rados.rados.0.smithi070.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-04-16T07:45:57.029 INFO:tasks.rados.rados.0.smithi070.stdout:6: done (15 left) 2024-04-16T07:45:57.029 INFO:tasks.rados.rados.0.smithi070.stdout:22: write initial oid 22 2024-04-16T07:45:57.029 INFO:tasks.rados.rados.0.smithi070.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-04-16T07:45:57.038 INFO:tasks.rados.rados.0.smithi070.stdout:22: writing smithi07031500-22 from 752947 to 1365882 tid 1 2024-04-16T07:45:57.046 INFO:tasks.rados.rados.0.smithi070.stdout:22: writing smithi07031500-22 from 1962710 to 2352947 tid 2 2024-04-16T07:45:57.046 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.046 INFO:tasks.rados.rados.0.smithi070.stdout:13: finishing write tid 3 to smithi07031500-13 2024-04-16T07:45:57.046 INFO:tasks.rados.rados.0.smithi070.stdout:13: finishing write tid 4 to smithi07031500-13 2024-04-16T07:45:57.046 INFO:tasks.rados.rados.0.smithi070.stdout:13: oid 13 updating version 0 to 1 2024-04-16T07:45:57.046 INFO:tasks.rados.rados.0.smithi070.stdout:13: oid 13 updating version 1 to 3 2024-04-16T07:45:57.046 INFO:tasks.rados.rados.0.smithi070.stdout:13: oid 13 version 3 is already newer than 2 2024-04-16T07:45:57.046 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 13 v 3 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-16T07:45:57.046 INFO:tasks.rados.rados.0.smithi070.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-04-16T07:45:57.046 INFO:tasks.rados.rados.0.smithi070.stdout:13: done (15 left) 2024-04-16T07:45:57.046 INFO:tasks.rados.rados.0.smithi070.stdout:23: write initial oid 23 2024-04-16T07:45:57.046 INFO:tasks.rados.rados.0.smithi070.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-04-16T07:45:57.055 INFO:tasks.rados.rados.0.smithi070.stdout:23: writing smithi07031500-23 from 769754 to 1457938 tid 1 2024-04-16T07:45:57.063 INFO:tasks.rados.rados.0.smithi070.stdout:23: writing smithi07031500-23 from 2021192 to 2369754 tid 2 2024-04-16T07:45:57.063 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.063 INFO:tasks.rados.rados.0.smithi070.stdout:12: finishing write tid 2 to smithi07031500-12 2024-04-16T07:45:57.063 INFO:tasks.rados.rados.0.smithi070.stdout:16: finishing write tid 2 to smithi07031500-16 2024-04-16T07:45:57.063 INFO:tasks.rados.rados.0.smithi070.stdout:17: finishing write tid 3 to smithi07031500-17 2024-04-16T07:45:57.063 INFO:tasks.rados.rados.0.smithi070.stdout:17: finishing write tid 4 to smithi07031500-17 2024-04-16T07:45:57.063 INFO:tasks.rados.rados.0.smithi070.stdout:17: oid 17 updating version 0 to 1 2024-04-16T07:45:57.063 INFO:tasks.rados.rados.0.smithi070.stdout:17: oid 17 updating version 1 to 3 2024-04-16T07:45:57.063 INFO:tasks.rados.rados.0.smithi070.stdout:17: oid 17 version 3 is already newer than 2 2024-04-16T07:45:57.063 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 17 v 3 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-16T07:45:57.064 INFO:tasks.rados.rados.0.smithi070.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-04-16T07:45:57.064 INFO:tasks.rados.rados.0.smithi070.stdout:15: finishing write tid 3 to smithi07031500-15 2024-04-16T07:45:57.064 INFO:tasks.rados.rados.0.smithi070.stdout:17: done (15 left) 2024-04-16T07:45:57.064 INFO:tasks.rados.rados.0.smithi070.stdout:24: write initial oid 24 2024-04-16T07:45:57.064 INFO:tasks.rados.rados.0.smithi070.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-04-16T07:45:57.070 INFO:tasks.rados.rados.0.smithi070.stdout:24: writing smithi07031500-24 from 786561 to 1266347 tid 1 2024-04-16T07:45:57.078 INFO:tasks.rados.rados.0.smithi070.stdout:24: writing smithi07031500-24 from 1796027 to 2263691 tid 2 2024-04-16T07:45:57.078 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.078 INFO:tasks.rados.rados.0.smithi070.stdout:10: finishing write tid 3 to smithi07031500-10 2024-04-16T07:45:57.079 INFO:tasks.rados.rados.0.smithi070.stdout:4: finishing write tid 1 to smithi07031500-4 2024-04-16T07:45:57.079 INFO:tasks.rados.rados.0.smithi070.stdout:15: finishing write tid 4 to smithi07031500-15 2024-04-16T07:45:57.079 INFO:tasks.rados.rados.0.smithi070.stdout:15: oid 15 updating version 0 to 3 2024-04-16T07:45:57.079 INFO:tasks.rados.rados.0.smithi070.stdout:15: oid 15 version 3 is already newer than 1 2024-04-16T07:45:57.079 INFO:tasks.rados.rados.0.smithi070.stdout:15: oid 15 version 3 is already newer than 2 2024-04-16T07:45:57.079 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-16T07:45:57.079 INFO:tasks.rados.rados.0.smithi070.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-04-16T07:45:57.079 INFO:tasks.rados.rados.0.smithi070.stdout:10: finishing write tid 4 to smithi07031500-10 2024-04-16T07:45:57.079 INFO:tasks.rados.rados.0.smithi070.stdout:10: oid 10 updating version 0 to 1 2024-04-16T07:45:57.079 INFO:tasks.rados.rados.0.smithi070.stdout:10: oid 10 updating version 1 to 3 2024-04-16T07:45:57.079 INFO:tasks.rados.rados.0.smithi070.stdout:10: oid 10 version 3 is already newer than 2 2024-04-16T07:45:57.079 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-16T07:45:57.079 INFO:tasks.rados.rados.0.smithi070.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-04-16T07:45:57.079 INFO:tasks.rados.rados.0.smithi070.stdout:7: finishing write tid 1 to smithi07031500-7 2024-04-16T07:45:57.079 INFO:tasks.rados.rados.0.smithi070.stdout:10: done (15 left) 2024-04-16T07:45:57.079 INFO:tasks.rados.rados.0.smithi070.stdout:15: done (14 left) 2024-04-16T07:45:57.079 INFO:tasks.rados.rados.0.smithi070.stdout:25: write initial oid 25 2024-04-16T07:45:57.080 INFO:tasks.rados.rados.0.smithi070.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-04-16T07:45:57.085 INFO:tasks.rados.rados.0.smithi070.stdout:25: writing smithi07031500-25 from 403368 to 958403 tid 1 2024-04-16T07:45:57.093 INFO:tasks.rados.rados.0.smithi070.stdout:25: writing smithi07031500-25 from 1454509 to 2182184 tid 2 2024-04-16T07:45:57.093 INFO:tasks.rados.rados.0.smithi070.stdout:26: write initial oid 26 2024-04-16T07:45:57.093 INFO:tasks.rados.rados.0.smithi070.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-04-16T07:45:57.100 INFO:tasks.rados.rados.0.smithi070.stdout:26: writing smithi07031500-26 from 420175 to 1050459 tid 1 2024-04-16T07:45:57.107 INFO:tasks.rados.rados.0.smithi070.stdout:26: writing smithi07031500-26 from 1796638 to 2267971 tid 2 2024-04-16T07:45:57.107 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.107 INFO:tasks.rados.rados.0.smithi070.stdout:16: finishing write tid 3 to smithi07031500-16 2024-04-16T07:45:57.107 INFO:tasks.rados.rados.0.smithi070.stdout:16: finishing write tid 4 to smithi07031500-16 2024-04-16T07:45:57.108 INFO:tasks.rados.rados.0.smithi070.stdout:16: oid 16 updating version 0 to 1 2024-04-16T07:45:57.108 INFO:tasks.rados.rados.0.smithi070.stdout:16: oid 16 updating version 1 to 3 2024-04-16T07:45:57.108 INFO:tasks.rados.rados.0.smithi070.stdout:16: oid 16 version 3 is already newer than 3 2024-04-16T07:45:57.108 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 16 v 3 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-16T07:45:57.108 INFO:tasks.rados.rados.0.smithi070.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-04-16T07:45:57.108 INFO:tasks.rados.rados.0.smithi070.stdout:9: finishing write tid 1 to smithi07031500-9 2024-04-16T07:45:57.108 INFO:tasks.rados.rados.0.smithi070.stdout:22: finishing write tid 1 to smithi07031500-22 2024-04-16T07:45:57.108 INFO:tasks.rados.rados.0.smithi070.stdout:20: finishing write tid 1 to smithi07031500-20 2024-04-16T07:45:57.108 INFO:tasks.rados.rados.0.smithi070.stdout:16: done (15 left) 2024-04-16T07:45:57.108 INFO:tasks.rados.rados.0.smithi070.stdout:27: write initial oid 27 2024-04-16T07:45:57.108 INFO:tasks.rados.rados.0.smithi070.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-04-16T07:45:57.116 INFO:tasks.rados.rados.0.smithi070.stdout:27: writing smithi07031500-27 from 436982 to 1142515 tid 1 2024-04-16T07:45:57.127 INFO:tasks.rados.rados.0.smithi070.stdout:27: writing smithi07031500-27 from 1855120 to 2436982 tid 2 2024-04-16T07:45:57.127 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.127 INFO:tasks.rados.rados.0.smithi070.stdout:18: finishing write tid 2 to smithi07031500-18 2024-04-16T07:45:57.127 INFO:tasks.rados.rados.0.smithi070.stdout:18: finishing write tid 3 to smithi07031500-18 2024-04-16T07:45:57.127 INFO:tasks.rados.rados.0.smithi070.stdout:18: finishing write tid 4 to smithi07031500-18 2024-04-16T07:45:57.127 INFO:tasks.rados.rados.0.smithi070.stdout:18: oid 18 updating version 0 to 3 2024-04-16T07:45:57.127 INFO:tasks.rados.rados.0.smithi070.stdout:18: oid 18 version 3 is already newer than 1 2024-04-16T07:45:57.127 INFO:tasks.rados.rados.0.smithi070.stdout:18: oid 18 version 3 is already newer than 3 2024-04-16T07:45:57.127 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 18 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-16T07:45:57.128 INFO:tasks.rados.rados.0.smithi070.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-04-16T07:45:57.128 INFO:tasks.rados.rados.0.smithi070.stdout:7: finishing write tid 2 to smithi07031500-7 2024-04-16T07:45:57.128 INFO:tasks.rados.rados.0.smithi070.stdout:12: finishing write tid 3 to smithi07031500-12 2024-04-16T07:45:57.128 INFO:tasks.rados.rados.0.smithi070.stdout:18: done (15 left) 2024-04-16T07:45:57.128 INFO:tasks.rados.rados.0.smithi070.stdout:28: write initial oid 28 2024-04-16T07:45:57.128 INFO:tasks.rados.rados.0.smithi070.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-04-16T07:45:57.137 INFO:tasks.rados.rados.0.smithi070.stdout:28: writing smithi07031500-28 from 453789 to 1234571 tid 1 2024-04-16T07:45:57.147 INFO:tasks.rados.rados.0.smithi070.stdout:28: writing smithi07031500-28 from 1913602 to 2388604 tid 2 2024-04-16T07:45:57.147 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.147 INFO:tasks.rados.rados.0.smithi070.stdout:22: finishing write tid 2 to smithi07031500-22 2024-04-16T07:45:57.147 INFO:tasks.rados.rados.0.smithi070.stdout:4: finishing write tid 2 to smithi07031500-4 2024-04-16T07:45:57.147 INFO:tasks.rados.rados.0.smithi070.stdout:7: finishing write tid 3 to smithi07031500-7 2024-04-16T07:45:57.147 INFO:tasks.rados.rados.0.smithi070.stdout:9: finishing write tid 2 to smithi07031500-9 2024-04-16T07:45:57.147 INFO:tasks.rados.rados.0.smithi070.stdout:12: finishing write tid 4 to smithi07031500-12 2024-04-16T07:45:57.147 INFO:tasks.rados.rados.0.smithi070.stdout:12: oid 12 updating version 0 to 1 2024-04-16T07:45:57.147 INFO:tasks.rados.rados.0.smithi070.stdout:12: oid 12 updating version 1 to 3 2024-04-16T07:45:57.147 INFO:tasks.rados.rados.0.smithi070.stdout:12: oid 12 version 3 is already newer than 2 2024-04-16T07:45:57.147 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-16T07:45:57.148 INFO:tasks.rados.rados.0.smithi070.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-04-16T07:45:57.148 INFO:tasks.rados.rados.0.smithi070.stdout:4: finishing write tid 3 to smithi07031500-4 2024-04-16T07:45:57.148 INFO:tasks.rados.rados.0.smithi070.stdout:12: done (15 left) 2024-04-16T07:45:57.148 INFO:tasks.rados.rados.0.smithi070.stdout:29: write initial oid 29 2024-04-16T07:45:57.148 INFO:tasks.rados.rados.0.smithi070.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-04-16T07:45:57.153 INFO:tasks.rados.rados.0.smithi070.stdout:29: writing smithi07031500-29 from 470596 to 926627 tid 1 2024-04-16T07:45:57.165 INFO:tasks.rados.rados.0.smithi070.stdout:29: writing smithi07031500-29 from 1572084 to 2190744 tid 2 2024-04-16T07:45:57.165 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.165 INFO:tasks.rados.rados.0.smithi070.stdout:7: finishing write tid 4 to smithi07031500-7 2024-04-16T07:45:57.166 INFO:tasks.rados.rados.0.smithi070.stdout:7: oid 7 updating version 0 to 6 2024-04-16T07:45:57.166 INFO:tasks.rados.rados.0.smithi070.stdout:7: oid 7 version 6 is already newer than 4 2024-04-16T07:45:57.166 INFO:tasks.rados.rados.0.smithi070.stdout:7: oid 7 version 6 is already newer than 6 2024-04-16T07:45:57.166 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 7 v 6 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-16T07:45:57.166 INFO:tasks.rados.rados.0.smithi070.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-04-16T07:45:57.166 INFO:tasks.rados.rados.0.smithi070.stdout:7: done (15 left) 2024-04-16T07:45:57.166 INFO:tasks.rados.rados.0.smithi070.stdout:30: write initial oid 30 2024-04-16T07:45:57.166 INFO:tasks.rados.rados.0.smithi070.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-04-16T07:45:57.173 INFO:tasks.rados.rados.0.smithi070.stdout:30: writing smithi07031500-30 from 487403 to 1018683 tid 1 2024-04-16T07:45:57.182 INFO:tasks.rados.rados.0.smithi070.stdout:30: writing smithi07031500-30 from 1514213 to 1992884 tid 2 2024-04-16T07:45:57.182 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.182 INFO:tasks.rados.rados.0.smithi070.stdout:4: finishing write tid 4 to smithi07031500-4 2024-04-16T07:45:57.182 INFO:tasks.rados.rados.0.smithi070.stdout:4: oid 4 updating version 0 to 9 2024-04-16T07:45:57.182 INFO:tasks.rados.rados.0.smithi070.stdout:4: oid 4 version 9 is already newer than 7 2024-04-16T07:45:57.182 INFO:tasks.rados.rados.0.smithi070.stdout:4: oid 4 version 9 is already newer than 9 2024-04-16T07:45:57.183 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 4 v 9 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-16T07:45:57.183 INFO:tasks.rados.rados.0.smithi070.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-04-16T07:45:57.183 INFO:tasks.rados.rados.0.smithi070.stdout:9: finishing write tid 3 to smithi07031500-9 2024-04-16T07:45:57.183 INFO:tasks.rados.rados.0.smithi070.stdout:4: done (15 left) 2024-04-16T07:45:57.183 INFO:tasks.rados.rados.0.smithi070.stdout:31: write initial oid 31 2024-04-16T07:45:57.183 INFO:tasks.rados.rados.0.smithi070.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-04-16T07:45:57.190 INFO:tasks.rados.rados.0.smithi070.stdout:31: writing smithi07031500-31 from 504210 to 1110739 tid 1 2024-04-16T07:45:57.200 INFO:tasks.rados.rados.0.smithi070.stdout:31: writing smithi07031500-31 from 1572695 to 2195024 tid 2 2024-04-16T07:45:57.200 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.200 INFO:tasks.rados.rados.0.smithi070.stdout:22: finishing write tid 3 to smithi07031500-22 2024-04-16T07:45:57.200 INFO:tasks.rados.rados.0.smithi070.stdout:9: finishing write tid 4 to smithi07031500-9 2024-04-16T07:45:57.200 INFO:tasks.rados.rados.0.smithi070.stdout:9: oid 9 updating version 0 to 6 2024-04-16T07:45:57.200 INFO:tasks.rados.rados.0.smithi070.stdout:9: oid 9 version 6 is already newer than 6 2024-04-16T07:45:57.200 INFO:tasks.rados.rados.0.smithi070.stdout:9: oid 9 version 6 is already newer than 4 2024-04-16T07:45:57.200 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 9 v 6 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-16T07:45:57.201 INFO:tasks.rados.rados.0.smithi070.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-04-16T07:45:57.201 INFO:tasks.rados.rados.0.smithi070.stdout:22: finishing write tid 4 to smithi07031500-22 2024-04-16T07:45:57.201 INFO:tasks.rados.rados.0.smithi070.stdout:22: oid 22 updating version 0 to 2 2024-04-16T07:45:57.201 INFO:tasks.rados.rados.0.smithi070.stdout:22: oid 22 version 2 is already newer than 1 2024-04-16T07:45:57.201 INFO:tasks.rados.rados.0.smithi070.stdout:22: oid 22 updating version 2 to 3 2024-04-16T07:45:57.201 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-16T07:45:57.201 INFO:tasks.rados.rados.0.smithi070.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-04-16T07:45:57.201 INFO:tasks.rados.rados.0.smithi070.stdout:9: done (15 left) 2024-04-16T07:45:57.201 INFO:tasks.rados.rados.0.smithi070.stdout:22: done (14 left) 2024-04-16T07:45:57.201 INFO:tasks.rados.rados.0.smithi070.stdout:32: write initial oid 32 2024-04-16T07:45:57.201 INFO:tasks.rados.rados.0.smithi070.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-04-16T07:45:57.211 INFO:tasks.rados.rados.0.smithi070.stdout:32: writing smithi07031500-32 from 521017 to 1319148 tid 1 2024-04-16T07:45:57.218 INFO:tasks.rados.rados.0.smithi070.stdout:32: writing smithi07031500-32 from 1747530 to 2229870 tid 2 2024-04-16T07:45:57.218 INFO:tasks.rados.rados.0.smithi070.stdout:33: write initial oid 33 2024-04-16T07:45:57.218 INFO:tasks.rados.rados.0.smithi070.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-04-16T07:45:57.226 INFO:tasks.rados.rados.0.smithi070.stdout:33: writing smithi07031500-33 from 537824 to 1011204 tid 1 2024-04-16T07:45:57.238 INFO:tasks.rados.rados.0.smithi070.stdout:33: writing smithi07031500-33 from 1806012 to 2432010 tid 2 2024-04-16T07:45:57.239 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.239 INFO:tasks.rados.rados.0.smithi070.stdout:20: finishing write tid 2 to smithi07031500-20 2024-04-16T07:45:57.239 INFO:tasks.rados.rados.0.smithi070.stdout:25: finishing write tid 1 to smithi07031500-25 2024-04-16T07:45:57.239 INFO:tasks.rados.rados.0.smithi070.stdout:20: finishing write tid 3 to smithi07031500-20 2024-04-16T07:45:57.239 INFO:tasks.rados.rados.0.smithi070.stdout:20: finishing write tid 4 to smithi07031500-20 2024-04-16T07:45:57.239 INFO:tasks.rados.rados.0.smithi070.stdout:20: oid 20 updating version 0 to 1 2024-04-16T07:45:57.239 INFO:tasks.rados.rados.0.smithi070.stdout:20: oid 20 updating version 1 to 3 2024-04-16T07:45:57.239 INFO:tasks.rados.rados.0.smithi070.stdout:20: oid 20 version 3 is already newer than 3 2024-04-16T07:45:57.239 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-16T07:45:57.239 INFO:tasks.rados.rados.0.smithi070.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-04-16T07:45:57.239 INFO:tasks.rados.rados.0.smithi070.stdout:24: finishing write tid 1 to smithi07031500-24 2024-04-16T07:45:57.239 INFO:tasks.rados.rados.0.smithi070.stdout:23: finishing write tid 1 to smithi07031500-23 2024-04-16T07:45:57.239 INFO:tasks.rados.rados.0.smithi070.stdout:11: finishing write tid 1 to smithi07031500-11 2024-04-16T07:45:57.239 INFO:tasks.rados.rados.0.smithi070.stdout:14: finishing write tid 1 to smithi07031500-14 2024-04-16T07:45:57.239 INFO:tasks.rados.rados.0.smithi070.stdout:20: done (15 left) 2024-04-16T07:45:57.239 INFO:tasks.rados.rados.0.smithi070.stdout:34: write initial oid 34 2024-04-16T07:45:57.240 INFO:tasks.rados.rados.0.smithi070.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-04-16T07:45:57.247 INFO:tasks.rados.rados.0.smithi070.stdout:34: writing smithi07031500-34 from 554631 to 1103260 tid 1 2024-04-16T07:45:57.256 INFO:tasks.rados.rados.0.smithi070.stdout:34: writing smithi07031500-34 from 1748141 to 2234150 tid 2 2024-04-16T07:45:57.256 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.256 INFO:tasks.rados.rados.0.smithi070.stdout:27: finishing write tid 1 to smithi07031500-27 2024-04-16T07:45:57.256 INFO:tasks.rados.rados.0.smithi070.stdout:25: finishing write tid 2 to smithi07031500-25 2024-04-16T07:45:57.256 INFO:tasks.rados.rados.0.smithi070.stdout:24: finishing write tid 2 to smithi07031500-24 2024-04-16T07:45:57.256 INFO:tasks.rados.rados.0.smithi070.stdout:29: finishing write tid 1 to smithi07031500-29 2024-04-16T07:45:57.256 INFO:tasks.rados.rados.0.smithi070.stdout:30: finishing write tid 1 to smithi07031500-30 2024-04-16T07:45:57.256 INFO:tasks.rados.rados.0.smithi070.stdout:11: finishing write tid 2 to smithi07031500-11 2024-04-16T07:45:57.256 INFO:tasks.rados.rados.0.smithi070.stdout:24: finishing write tid 3 to smithi07031500-24 2024-04-16T07:45:57.259 INFO:tasks.rados.rados.0.smithi070.stdout:24: finishing write tid 4 to smithi07031500-24 2024-04-16T07:45:57.259 INFO:tasks.rados.rados.0.smithi070.stdout:24: oid 24 updating version 0 to 1 2024-04-16T07:45:57.259 INFO:tasks.rados.rados.0.smithi070.stdout:24: oid 24 updating version 1 to 3 2024-04-16T07:45:57.259 INFO:tasks.rados.rados.0.smithi070.stdout:24: oid 24 version 3 is already newer than 3 2024-04-16T07:45:57.259 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 24 v 3 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-16T07:45:57.259 INFO:tasks.rados.rados.0.smithi070.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-04-16T07:45:57.259 INFO:tasks.rados.rados.0.smithi070.stdout:24: done (15 left) 2024-04-16T07:45:57.259 INFO:tasks.rados.rados.0.smithi070.stdout:35: write initial oid 35 2024-04-16T07:45:57.259 INFO:tasks.rados.rados.0.smithi070.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-04-16T07:45:57.266 INFO:tasks.rados.rados.0.smithi070.stdout:35: writing smithi07031500-35 from 571438 to 1195316 tid 1 2024-04-16T07:45:57.274 INFO:tasks.rados.rados.0.smithi070.stdout:35: writing smithi07031500-35 from 1806623 to 2436290 tid 2 2024-04-16T07:45:57.274 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.274 INFO:tasks.rados.rados.0.smithi070.stdout:11: finishing write tid 3 to smithi07031500-11 2024-04-16T07:45:57.274 INFO:tasks.rados.rados.0.smithi070.stdout:23: finishing write tid 2 to smithi07031500-23 2024-04-16T07:45:57.274 INFO:tasks.rados.rados.0.smithi070.stdout:27: finishing write tid 2 to smithi07031500-27 2024-04-16T07:45:57.274 INFO:tasks.rados.rados.0.smithi070.stdout:11: finishing write tid 4 to smithi07031500-11 2024-04-16T07:45:57.274 INFO:tasks.rados.rados.0.smithi070.stdout:11: oid 11 updating version 0 to 9 2024-04-16T07:45:57.275 INFO:tasks.rados.rados.0.smithi070.stdout:11: oid 11 version 9 is already newer than 7 2024-04-16T07:45:57.275 INFO:tasks.rados.rados.0.smithi070.stdout:11: oid 11 version 9 is already newer than 9 2024-04-16T07:45:57.275 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 11 v 9 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-16T07:45:57.275 INFO:tasks.rados.rados.0.smithi070.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-04-16T07:45:57.275 INFO:tasks.rados.rados.0.smithi070.stdout:11: done (15 left) 2024-04-16T07:45:57.275 INFO:tasks.rados.rados.0.smithi070.stdout:36: write initial oid 36 2024-04-16T07:45:57.275 INFO:tasks.rados.rados.0.smithi070.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-04-16T07:45:57.284 INFO:tasks.rados.rados.0.smithi070.stdout:36: writing smithi07031500-36 from 588245 to 1287372 tid 1 2024-04-16T07:45:57.292 INFO:tasks.rados.rados.0.smithi070.stdout:36: writing smithi07031500-36 from 1865105 to 2354783 tid 2 2024-04-16T07:45:57.292 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.293 INFO:tasks.rados.rados.0.smithi070.stdout:14: finishing write tid 2 to smithi07031500-14 2024-04-16T07:45:57.293 INFO:tasks.rados.rados.0.smithi070.stdout:31: finishing write tid 1 to smithi07031500-31 2024-04-16T07:45:57.293 INFO:tasks.rados.rados.0.smithi070.stdout:23: finishing write tid 3 to smithi07031500-23 2024-04-16T07:45:57.293 INFO:tasks.rados.rados.0.smithi070.stdout:23: finishing write tid 4 to smithi07031500-23 2024-04-16T07:45:57.293 INFO:tasks.rados.rados.0.smithi070.stdout:23: oid 23 updating version 0 to 4 2024-04-16T07:45:57.293 INFO:tasks.rados.rados.0.smithi070.stdout:23: oid 23 updating version 4 to 6 2024-04-16T07:45:57.293 INFO:tasks.rados.rados.0.smithi070.stdout:23: oid 23 version 6 is already newer than 6 2024-04-16T07:45:57.293 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-16T07:45:57.293 INFO:tasks.rados.rados.0.smithi070.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-04-16T07:45:57.293 INFO:tasks.rados.rados.0.smithi070.stdout:14: finishing write tid 3 to smithi07031500-14 2024-04-16T07:45:57.293 INFO:tasks.rados.rados.0.smithi070.stdout:14: finishing write tid 4 to smithi07031500-14 2024-04-16T07:45:57.293 INFO:tasks.rados.rados.0.smithi070.stdout:14: oid 14 updating version 0 to 12 2024-04-16T07:45:57.293 INFO:tasks.rados.rados.0.smithi070.stdout:14: oid 14 version 12 is already newer than 12 2024-04-16T07:45:57.293 INFO:tasks.rados.rados.0.smithi070.stdout:14: oid 14 version 12 is already newer than 10 2024-04-16T07:45:57.293 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 14 v 12 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-16T07:45:57.294 INFO:tasks.rados.rados.0.smithi070.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-04-16T07:45:57.294 INFO:tasks.rados.rados.0.smithi070.stdout:14: done (15 left) 2024-04-16T07:45:57.294 INFO:tasks.rados.rados.0.smithi070.stdout:23: done (14 left) 2024-04-16T07:45:57.294 INFO:tasks.rados.rados.0.smithi070.stdout:37: write initial oid 37 2024-04-16T07:45:57.294 INFO:tasks.rados.rados.0.smithi070.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-04-16T07:45:57.302 INFO:tasks.rados.rados.0.smithi070.stdout:37: writing smithi07031500-37 from 605052 to 1379428 tid 1 2024-04-16T07:45:57.309 INFO:tasks.rados.rados.0.smithi070.stdout:37: writing smithi07031500-37 from 1923587 to 2556923 tid 2 2024-04-16T07:45:57.309 INFO:tasks.rados.rados.0.smithi070.stdout:38: write initial oid 38 2024-04-16T07:45:57.309 INFO:tasks.rados.rados.0.smithi070.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-04-16T07:45:57.317 INFO:tasks.rados.rados.0.smithi070.stdout:38: writing smithi07031500-38 from 621859 to 1071484 tid 1 2024-04-16T07:45:57.329 INFO:tasks.rados.rados.0.smithi070.stdout:38: writing smithi07031500-38 from 1865716 to 2621859 tid 2 2024-04-16T07:45:57.330 INFO:tasks.rados.rados.0.smithi070.stdout:25: finishing write tid 3 to smithi07031500-25 2024-04-16T07:45:57.330 INFO:tasks.rados.rados.0.smithi070.stdout:25: finishing write tid 4 to smithi07031500-25 2024-04-16T07:45:57.330 INFO:tasks.rados.rados.0.smithi070.stdout:25: oid 25 updating version 0 to 3 2024-04-16T07:45:57.330 INFO:tasks.rados.rados.0.smithi070.stdout:25: oid 25 version 3 is already newer than 1 2024-04-16T07:45:57.330 INFO:tasks.rados.rados.0.smithi070.stdout:25: oid 25 version 3 is already newer than 2 2024-04-16T07:45:57.331 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 25 v 3 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-16T07:45:57.331 INFO:tasks.rados.rados.0.smithi070.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-04-16T07:45:57.331 INFO:tasks.rados.rados.0.smithi070.stdout:30: finishing write tid 2 to smithi07031500-30 2024-04-16T07:45:57.331 INFO:tasks.rados.rados.0.smithi070.stdout:31: finishing write tid 2 to smithi07031500-31 2024-04-16T07:45:57.331 INFO:tasks.rados.rados.0.smithi070.stdout:25: done (15 left) 2024-04-16T07:45:57.331 INFO:tasks.rados.rados.0.smithi070.stdout:39: write initial oid 39 2024-04-16T07:45:57.331 INFO:tasks.rados.rados.0.smithi070.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-04-16T07:45:57.336 INFO:tasks.rados.rados.0.smithi070.stdout:39: writing smithi07031500-39 from 638666 to 1163540 tid 1 2024-04-16T07:45:57.348 INFO:tasks.rados.rados.0.smithi070.stdout:39: writing smithi07031500-39 from 1924198 to 2561203 tid 2 2024-04-16T07:45:57.348 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.348 INFO:tasks.rados.rados.0.smithi070.stdout:34: finishing write tid 1 to smithi07031500-34 2024-04-16T07:45:57.348 INFO:tasks.rados.rados.0.smithi070.stdout:32: finishing write tid 1 to smithi07031500-32 2024-04-16T07:45:57.348 INFO:tasks.rados.rados.0.smithi070.stdout:31: finishing write tid 3 to smithi07031500-31 2024-04-16T07:45:57.348 INFO:tasks.rados.rados.0.smithi070.stdout:31: finishing write tid 4 to smithi07031500-31 2024-04-16T07:45:57.348 INFO:tasks.rados.rados.0.smithi070.stdout:31: oid 31 updating version 0 to 5 2024-04-16T07:45:57.349 INFO:tasks.rados.rados.0.smithi070.stdout:31: oid 31 version 5 is already newer than 4 2024-04-16T07:45:57.349 INFO:tasks.rados.rados.0.smithi070.stdout:31: oid 31 updating version 5 to 6 2024-04-16T07:45:57.349 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-16T07:45:57.349 INFO:tasks.rados.rados.0.smithi070.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-04-16T07:45:57.349 INFO:tasks.rados.rados.0.smithi070.stdout:31: done (15 left) 2024-04-16T07:45:57.349 INFO:tasks.rados.rados.0.smithi070.stdout:40: write initial oid 40 2024-04-16T07:45:57.349 INFO:tasks.rados.rados.0.smithi070.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-04-16T07:45:57.355 INFO:tasks.rados.rados.0.smithi070.stdout:40: writing smithi07031500-40 from 655473 to 1371949 tid 1 2024-04-16T07:45:57.363 INFO:tasks.rados.rados.0.smithi070.stdout:40: writing smithi07031500-40 from 2099033 to 2655473 tid 2 2024-04-16T07:45:57.363 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.364 INFO:tasks.rados.rados.0.smithi070.stdout:27: finishing write tid 3 to smithi07031500-27 2024-04-16T07:45:57.364 INFO:tasks.rados.rados.0.smithi070.stdout:27: finishing write tid 4 to smithi07031500-27 2024-04-16T07:45:57.364 INFO:tasks.rados.rados.0.smithi070.stdout:27: oid 27 updating version 0 to 4 2024-04-16T07:45:57.364 INFO:tasks.rados.rados.0.smithi070.stdout:27: oid 27 updating version 4 to 6 2024-04-16T07:45:57.364 INFO:tasks.rados.rados.0.smithi070.stdout:27: oid 27 version 6 is already newer than 5 2024-04-16T07:45:57.364 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-16T07:45:57.364 INFO:tasks.rados.rados.0.smithi070.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-04-16T07:45:57.364 INFO:tasks.rados.rados.0.smithi070.stdout:29: finishing write tid 2 to smithi07031500-29 2024-04-16T07:45:57.364 INFO:tasks.rados.rados.0.smithi070.stdout:27: done (15 left) 2024-04-16T07:45:57.364 INFO:tasks.rados.rados.0.smithi070.stdout:41: write initial oid 41 2024-04-16T07:45:57.364 INFO:tasks.rados.rados.0.smithi070.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-04-16T07:45:57.371 INFO:tasks.rados.rados.0.smithi070.stdout:41: writing smithi07031500-41 from 672280 to 1464005 tid 1 2024-04-16T07:45:57.381 INFO:tasks.rados.rados.0.smithi070.stdout:41: writing smithi07031500-41 from 2157515 to 2672280 tid 2 2024-04-16T07:45:57.381 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.381 INFO:tasks.rados.rados.0.smithi070.stdout:19: finishing write tid 1 to smithi07031500-19 2024-04-16T07:45:57.381 INFO:tasks.rados.rados.0.smithi070.stdout:33: finishing write tid 1 to smithi07031500-33 2024-04-16T07:45:57.381 INFO:tasks.rados.rados.0.smithi070.stdout:21: finishing write tid 1 to smithi07031500-21 2024-04-16T07:45:57.385 INFO:tasks.rados.rados.0.smithi070.stdout:37: finishing write tid 1 to smithi07031500-37 2024-04-16T07:45:57.392 INFO:tasks.rados.rados.0.smithi070.stdout:30: finishing write tid 3 to smithi07031500-30 2024-04-16T07:45:57.394 INFO:tasks.rados.rados.0.smithi070.stdout:30: finishing write tid 4 to smithi07031500-30 2024-04-16T07:45:57.394 INFO:tasks.rados.rados.0.smithi070.stdout:30: oid 30 updating version 0 to 5 2024-04-16T07:45:57.394 INFO:tasks.rados.rados.0.smithi070.stdout:30: oid 30 version 5 is already newer than 4 2024-04-16T07:45:57.394 INFO:tasks.rados.rados.0.smithi070.stdout:30: oid 30 updating version 5 to 6 2024-04-16T07:45:57.394 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-16T07:45:57.394 INFO:tasks.rados.rados.0.smithi070.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-04-16T07:45:57.394 INFO:tasks.rados.rados.0.smithi070.stdout:30: done (15 left) 2024-04-16T07:45:57.394 INFO:tasks.rados.rados.0.smithi070.stdout:42: write initial oid 42 2024-04-16T07:45:57.395 INFO:tasks.rados.rados.0.smithi070.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-04-16T07:45:57.402 INFO:tasks.rados.rados.0.smithi070.stdout:42: writing smithi07031500-42 from 689087 to 1156061 tid 1 2024-04-16T07:45:57.414 INFO:tasks.rados.rados.0.smithi070.stdout:42: writing smithi07031500-42 from 1699644 to 2483976 tid 2 2024-04-16T07:45:57.414 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.414 INFO:tasks.rados.rados.0.smithi070.stdout:34: finishing write tid 2 to smithi07031500-34 2024-04-16T07:45:57.414 INFO:tasks.rados.rados.0.smithi070.stdout:29: finishing write tid 3 to smithi07031500-29 2024-04-16T07:45:57.414 INFO:tasks.rados.rados.0.smithi070.stdout:19: finishing write tid 2 to smithi07031500-19 2024-04-16T07:45:57.415 INFO:tasks.rados.rados.0.smithi070.stdout:34: finishing write tid 3 to smithi07031500-34 2024-04-16T07:45:57.415 INFO:tasks.rados.rados.0.smithi070.stdout:29: finishing write tid 4 to smithi07031500-29 2024-04-16T07:45:57.415 INFO:tasks.rados.rados.0.smithi070.stdout:29: oid 29 updating version 0 to 4 2024-04-16T07:45:57.415 INFO:tasks.rados.rados.0.smithi070.stdout:29: oid 29 updating version 4 to 5 2024-04-16T07:45:57.415 INFO:tasks.rados.rados.0.smithi070.stdout:29: oid 29 updating version 5 to 6 2024-04-16T07:45:57.415 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 29 v 6 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-16T07:45:57.415 INFO:tasks.rados.rados.0.smithi070.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-04-16T07:45:57.415 INFO:tasks.rados.rados.0.smithi070.stdout:32: finishing write tid 2 to smithi07031500-32 2024-04-16T07:45:57.415 INFO:tasks.rados.rados.0.smithi070.stdout:29: done (15 left) 2024-04-16T07:45:57.415 INFO:tasks.rados.rados.0.smithi070.stdout:43: write initial oid 43 2024-04-16T07:45:57.415 INFO:tasks.rados.rados.0.smithi070.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-04-16T07:45:57.423 INFO:tasks.rados.rados.0.smithi070.stdout:43: writing smithi07031500-43 from 705894 to 1248117 tid 1 2024-04-16T07:45:57.433 INFO:tasks.rados.rados.0.smithi070.stdout:43: writing smithi07031500-43 from 1758126 to 2402469 tid 2 2024-04-16T07:45:57.433 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.433 INFO:tasks.rados.rados.0.smithi070.stdout:34: finishing write tid 4 to smithi07031500-34 2024-04-16T07:45:57.433 INFO:tasks.rados.rados.0.smithi070.stdout:34: oid 34 updating version 0 to 6 2024-04-16T07:45:57.433 INFO:tasks.rados.rados.0.smithi070.stdout:34: oid 34 version 6 is already newer than 6 2024-04-16T07:45:57.433 INFO:tasks.rados.rados.0.smithi070.stdout:34: oid 34 version 6 is already newer than 4 2024-04-16T07:45:57.433 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 34 v 6 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-16T07:45:57.434 INFO:tasks.rados.rados.0.smithi070.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-04-16T07:45:57.434 INFO:tasks.rados.rados.0.smithi070.stdout:34: done (15 left) 2024-04-16T07:45:57.434 INFO:tasks.rados.rados.0.smithi070.stdout:44: write initial oid 44 2024-04-16T07:45:57.434 INFO:tasks.rados.rados.0.smithi070.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-04-16T07:45:57.443 INFO:tasks.rados.rados.0.smithi070.stdout:44: writing smithi07031500-44 from 722701 to 1340173 tid 1 2024-04-16T07:45:57.454 INFO:tasks.rados.rados.0.smithi070.stdout:44: writing smithi07031500-44 from 1816608 to 2604609 tid 2 2024-04-16T07:45:57.455 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.455 INFO:tasks.rados.rados.0.smithi070.stdout:32: finishing write tid 3 to smithi07031500-32 2024-04-16T07:45:57.455 INFO:tasks.rados.rados.0.smithi070.stdout:19: finishing write tid 3 to smithi07031500-19 2024-04-16T07:45:57.455 INFO:tasks.rados.rados.0.smithi070.stdout:21: finishing write tid 2 to smithi07031500-21 2024-04-16T07:45:57.455 INFO:tasks.rados.rados.0.smithi070.stdout:32: finishing write tid 4 to smithi07031500-32 2024-04-16T07:45:57.455 INFO:tasks.rados.rados.0.smithi070.stdout:32: oid 32 updating version 0 to 6 2024-04-16T07:45:57.455 INFO:tasks.rados.rados.0.smithi070.stdout:32: oid 32 version 6 is already newer than 6 2024-04-16T07:45:57.455 INFO:tasks.rados.rados.0.smithi070.stdout:32: oid 32 version 6 is already newer than 4 2024-04-16T07:45:57.455 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-16T07:45:57.455 INFO:tasks.rados.rados.0.smithi070.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-04-16T07:45:57.455 INFO:tasks.rados.rados.0.smithi070.stdout:21: finishing write tid 3 to smithi07031500-21 2024-04-16T07:45:57.455 INFO:tasks.rados.rados.0.smithi070.stdout:21: finishing write tid 4 to smithi07031500-21 2024-04-16T07:45:57.455 INFO:tasks.rados.rados.0.smithi070.stdout:21: oid 21 updating version 0 to 13 2024-04-16T07:45:57.455 INFO:tasks.rados.rados.0.smithi070.stdout:21: oid 21 updating version 13 to 15 2024-04-16T07:45:57.455 INFO:tasks.rados.rados.0.smithi070.stdout:21: oid 21 version 15 is already newer than 15 2024-04-16T07:45:57.455 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 21 v 15 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-16T07:45:57.456 INFO:tasks.rados.rados.0.smithi070.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-04-16T07:45:57.456 INFO:tasks.rados.rados.0.smithi070.stdout:19: finishing write tid 4 to smithi07031500-19 2024-04-16T07:45:57.456 INFO:tasks.rados.rados.0.smithi070.stdout:19: oid 19 updating version 0 to 12 2024-04-16T07:45:57.456 INFO:tasks.rados.rados.0.smithi070.stdout:19: oid 19 version 12 is already newer than 12 2024-04-16T07:45:57.456 INFO:tasks.rados.rados.0.smithi070.stdout:19: oid 19 version 12 is already newer than 10 2024-04-16T07:45:57.456 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 19 v 12 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-16T07:45:57.456 INFO:tasks.rados.rados.0.smithi070.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-04-16T07:45:57.456 INFO:tasks.rados.rados.0.smithi070.stdout:19: done (15 left) 2024-04-16T07:45:57.456 INFO:tasks.rados.rados.0.smithi070.stdout:21: done (14 left) 2024-04-16T07:45:57.456 INFO:tasks.rados.rados.0.smithi070.stdout:32: done (13 left) 2024-04-16T07:45:57.456 INFO:tasks.rados.rados.0.smithi070.stdout:45: write initial oid 45 2024-04-16T07:45:57.456 INFO:tasks.rados.rados.0.smithi070.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-04-16T07:45:57.465 INFO:tasks.rados.rados.0.smithi070.stdout:45: writing smithi07031500-45 from 739508 to 1432229 tid 1 2024-04-16T07:45:57.475 INFO:tasks.rados.rados.0.smithi070.stdout:45: writing smithi07031500-45 from 1875090 to 2523102 tid 2 2024-04-16T07:45:57.475 INFO:tasks.rados.rados.0.smithi070.stdout:46: write initial oid 46 2024-04-16T07:45:57.475 INFO:tasks.rados.rados.0.smithi070.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-04-16T07:45:57.485 INFO:tasks.rados.rados.0.smithi070.stdout:46: writing smithi07031500-46 from 756315 to 1524285 tid 1 2024-04-16T07:45:57.494 INFO:tasks.rados.rados.0.smithi070.stdout:46: writing smithi07031500-46 from 1933572 to 2725242 tid 2 2024-04-16T07:45:57.494 INFO:tasks.rados.rados.0.smithi070.stdout:47: write initial oid 47 2024-04-16T07:45:57.494 INFO:tasks.rados.rados.0.smithi070.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-04-16T07:45:57.505 INFO:tasks.rados.rados.0.smithi070.stdout:47: writing smithi07031500-47 from 773122 to 1332694 tid 1 2024-04-16T07:45:57.514 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:45:57.517 INFO:tasks.rados.rados.0.smithi070.stdout:47: writing smithi07031500-47 from 1992054 to 2643735 tid 2 2024-04-16T07:45:57.517 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.517 INFO:tasks.rados.rados.0.smithi070.stdout:33: finishing write tid 2 to smithi07031500-33 2024-04-16T07:45:57.517 INFO:tasks.rados.rados.0.smithi070.stdout:40: finishing write tid 1 to smithi07031500-40 2024-04-16T07:45:57.517 INFO:tasks.rados.rados.0.smithi070.stdout:37: finishing write tid 2 to smithi07031500-37 2024-04-16T07:45:57.517 INFO:tasks.rados.rados.0.smithi070.stdout:35: finishing write tid 1 to smithi07031500-35 2024-04-16T07:45:57.517 INFO:tasks.rados.rados.0.smithi070.stdout:37: finishing write tid 3 to smithi07031500-37 2024-04-16T07:45:57.517 INFO:tasks.rados.rados.0.smithi070.stdout:37: finishing write tid 4 to smithi07031500-37 2024-04-16T07:45:57.517 INFO:tasks.rados.rados.0.smithi070.stdout:37: oid 37 updating version 0 to 6 2024-04-16T07:45:57.518 INFO:tasks.rados.rados.0.smithi070.stdout:37: oid 37 version 6 is already newer than 6 2024-04-16T07:45:57.518 INFO:tasks.rados.rados.0.smithi070.stdout:37: oid 37 version 6 is already newer than 4 2024-04-16T07:45:57.518 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-16T07:45:57.518 INFO:tasks.rados.rados.0.smithi070.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-04-16T07:45:57.518 INFO:tasks.rados.rados.0.smithi070.stdout:42: finishing write tid 1 to smithi07031500-42 2024-04-16T07:45:57.518 INFO:tasks.rados.rados.0.smithi070.stdout:36: finishing write tid 1 to smithi07031500-36 2024-04-16T07:45:57.518 INFO:tasks.rados.rados.0.smithi070.stdout:26: finishing write tid 1 to smithi07031500-26 2024-04-16T07:45:57.518 INFO:tasks.rados.rados.0.smithi070.stdout:37: done (15 left) 2024-04-16T07:45:57.518 INFO:tasks.rados.rados.0.smithi070.stdout:48: write initial oid 48 2024-04-16T07:45:57.518 INFO:tasks.rados.rados.0.smithi070.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-04-16T07:45:57.527 INFO:tasks.rados.rados.0.smithi070.stdout:48: writing smithi07031500-48 from 789929 to 1424750 tid 1 2024-04-16T07:45:57.539 INFO:tasks.rados.rados.0.smithi070.stdout:48: writing smithi07031500-48 from 2050536 to 2789929 tid 2 2024-04-16T07:45:57.539 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.539 INFO:tasks.rados.rados.0.smithi070.stdout:40: finishing write tid 2 to smithi07031500-40 2024-04-16T07:45:57.539 INFO:tasks.rados.rados.0.smithi070.stdout:33: finishing write tid 3 to smithi07031500-33 2024-04-16T07:45:57.539 INFO:tasks.rados.rados.0.smithi070.stdout:45: finishing write tid 1 to smithi07031500-45 2024-04-16T07:45:57.540 INFO:tasks.rados.rados.0.smithi070.stdout:33: finishing write tid 4 to smithi07031500-33 2024-04-16T07:45:57.540 INFO:tasks.rados.rados.0.smithi070.stdout:33: oid 33 updating version 0 to 9 2024-04-16T07:45:57.540 INFO:tasks.rados.rados.0.smithi070.stdout:33: oid 33 version 9 is already newer than 7 2024-04-16T07:45:57.540 INFO:tasks.rados.rados.0.smithi070.stdout:33: oid 33 version 9 is already newer than 8 2024-04-16T07:45:57.540 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 33 v 9 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-16T07:45:57.540 INFO:tasks.rados.rados.0.smithi070.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-04-16T07:45:57.540 INFO:tasks.rados.rados.0.smithi070.stdout:43: finishing write tid 1 to smithi07031500-43 2024-04-16T07:45:57.540 INFO:tasks.rados.rados.0.smithi070.stdout:33: done (15 left) 2024-04-16T07:45:57.541 INFO:tasks.rados.rados.0.smithi070.stdout:49: write initial oid 49 2024-04-16T07:45:57.541 INFO:tasks.rados.rados.0.smithi070.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-04-16T07:45:57.549 INFO:tasks.rados.rados.0.smithi070.stdout:49: writing smithi07031500-49 from 406736 to 1116806 tid 1 2024-04-16T07:45:57.556 INFO:tasks.rados.rados.0.smithi070.stdout:49: writing smithi07031500-49 from 1709018 to 2364368 tid 2 2024-04-16T07:45:57.557 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.557 INFO:tasks.rados.rados.0.smithi070.stdout:35: finishing write tid 2 to smithi07031500-35 2024-04-16T07:45:57.564 INFO:tasks.rados.rados.0.smithi070.stdout:35: finishing write tid 3 to smithi07031500-35 2024-04-16T07:45:57.572 INFO:tasks.rados.rados.0.smithi070.stdout:35: finishing write tid 4 to smithi07031500-35 2024-04-16T07:45:57.572 INFO:tasks.rados.rados.0.smithi070.stdout:35: oid 35 updating version 0 to 9 2024-04-16T07:45:57.572 INFO:tasks.rados.rados.0.smithi070.stdout:35: oid 35 version 9 is already newer than 9 2024-04-16T07:45:57.572 INFO:tasks.rados.rados.0.smithi070.stdout:35: oid 35 version 9 is already newer than 7 2024-04-16T07:45:57.572 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 35 v 9 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-16T07:45:57.572 INFO:tasks.rados.rados.0.smithi070.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-04-16T07:45:57.572 INFO:tasks.rados.rados.0.smithi070.stdout:35: done (15 left) 2024-04-16T07:45:57.572 INFO:tasks.rados.rados.0.smithi070.stdout:50: write initial oid 50 2024-04-16T07:45:57.572 INFO:tasks.rados.rados.0.smithi070.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-04-16T07:45:57.578 INFO:tasks.rados.rados.0.smithi070.stdout:50: writing smithi07031500-50 from 423543 to 1208862 tid 1 2024-04-16T07:45:57.587 INFO:tasks.rados.rados.0.smithi070.stdout:50: writing smithi07031500-50 from 1767500 to 2566508 tid 2 2024-04-16T07:45:57.587 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.587 INFO:tasks.rados.rados.0.smithi070.stdout:36: finishing write tid 2 to smithi07031500-36 2024-04-16T07:45:57.587 INFO:tasks.rados.rados.0.smithi070.stdout:40: finishing write tid 3 to smithi07031500-40 2024-04-16T07:45:57.587 INFO:tasks.rados.rados.0.smithi070.stdout:40: finishing write tid 4 to smithi07031500-40 2024-04-16T07:45:57.588 INFO:tasks.rados.rados.0.smithi070.stdout:40: oid 40 updating version 0 to 5 2024-04-16T07:45:57.588 INFO:tasks.rados.rados.0.smithi070.stdout:40: oid 40 updating version 5 to 6 2024-04-16T07:45:57.588 INFO:tasks.rados.rados.0.smithi070.stdout:40: oid 40 version 6 is already newer than 4 2024-04-16T07:45:57.588 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 40 v 6 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-16T07:45:57.588 INFO:tasks.rados.rados.0.smithi070.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-04-16T07:45:57.588 INFO:tasks.rados.rados.0.smithi070.stdout:41: finishing write tid 1 to smithi07031500-41 2024-04-16T07:45:57.588 INFO:tasks.rados.rados.0.smithi070.stdout:42: finishing write tid 2 to smithi07031500-42 2024-04-16T07:45:57.588 INFO:tasks.rados.rados.0.smithi070.stdout:40: done (15 left) 2024-04-16T07:45:57.588 INFO:tasks.rados.rados.0.smithi070.stdout:51: write initial oid 51 2024-04-16T07:45:57.588 INFO:tasks.rados.rados.0.smithi070.stdout:51: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2024-04-16T07:45:57.593 INFO:tasks.rados.rados.0.smithi070.stdout:51: writing smithi07031500-51 from 440350 to 900918 tid 1 2024-04-16T07:45:57.601 INFO:tasks.rados.rados.0.smithi070.stdout:51: writing smithi07031500-51 from 1309629 to 1852295 tid 2 2024-04-16T07:45:57.609 INFO:tasks.rados.rados.0.smithi070.stdout:51: writing smithi07031500-51 from 2323973 to 2840350 tid 3 2024-04-16T07:45:57.609 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.609 INFO:tasks.rados.rados.0.smithi070.stdout:26: finishing write tid 2 to smithi07031500-26 2024-04-16T07:45:57.609 INFO:tasks.rados.rados.0.smithi070.stdout:26: finishing write tid 3 to smithi07031500-26 2024-04-16T07:45:57.609 INFO:tasks.rados.rados.0.smithi070.stdout:45: finishing write tid 2 to smithi07031500-45 2024-04-16T07:45:57.609 INFO:tasks.rados.rados.0.smithi070.stdout:44: finishing write tid 1 to smithi07031500-44 2024-04-16T07:45:57.609 INFO:tasks.rados.rados.0.smithi070.stdout:43: finishing write tid 2 to smithi07031500-43 2024-04-16T07:45:57.609 INFO:tasks.rados.rados.0.smithi070.stdout:26: finishing write tid 4 to smithi07031500-26 2024-04-16T07:45:57.609 INFO:tasks.rados.rados.0.smithi070.stdout:26: oid 26 updating version 0 to 13 2024-04-16T07:45:57.609 INFO:tasks.rados.rados.0.smithi070.stdout:26: oid 26 updating version 13 to 15 2024-04-16T07:45:57.609 INFO:tasks.rados.rados.0.smithi070.stdout:26: oid 26 version 15 is already newer than 15 2024-04-16T07:45:57.610 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 26 v 15 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-16T07:45:57.610 INFO:tasks.rados.rados.0.smithi070.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-04-16T07:45:57.610 INFO:tasks.rados.rados.0.smithi070.stdout:26: done (15 left) 2024-04-16T07:45:57.610 INFO:tasks.rados.rados.0.smithi070.stdout:52: write initial oid 52 2024-04-16T07:45:57.610 INFO:tasks.rados.rados.0.smithi070.stdout:52: seq_num 51 ranges {457157=535817,1768111=402677} 2024-04-16T07:45:57.614 INFO:tasks.rados.rados.0.smithi070.stdout:52: writing smithi07031500-52 from 457157 to 992974 tid 1 2024-04-16T07:45:57.621 INFO:tasks.rados.rados.0.smithi070.stdout:52: writing smithi07031500-52 from 1768111 to 2170788 tid 2 2024-04-16T07:45:57.621 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.621 INFO:tasks.rados.rados.0.smithi070.stdout:36: finishing write tid 3 to smithi07031500-36 2024-04-16T07:45:57.623 INFO:tasks.rados.rados.0.smithi070.stdout:42: finishing write tid 3 to smithi07031500-42 2024-04-16T07:45:57.631 INFO:tasks.rados.rados.0.smithi070.stdout:42: finishing write tid 4 to smithi07031500-42 2024-04-16T07:45:57.631 INFO:tasks.rados.rados.0.smithi070.stdout:42: oid 42 updating version 0 to 8 2024-04-16T07:45:57.631 INFO:tasks.rados.rados.0.smithi070.stdout:42: oid 42 version 8 is already newer than 7 2024-04-16T07:45:57.631 INFO:tasks.rados.rados.0.smithi070.stdout:42: oid 42 updating version 8 to 9 2024-04-16T07:45:57.631 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 42 v 9 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-16T07:45:57.631 INFO:tasks.rados.rados.0.smithi070.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-04-16T07:45:57.632 INFO:tasks.rados.rados.0.smithi070.stdout:42: done (15 left) 2024-04-16T07:45:57.632 INFO:tasks.rados.rados.0.smithi070.stdout:53: write initial oid 53 2024-04-16T07:45:57.632 INFO:tasks.rados.rados.0.smithi070.stdout:53: seq_num 52 ranges {473964=611066,1826593=546335,2778819=95145} 2024-04-16T07:45:57.638 INFO:tasks.rados.rados.0.smithi070.stdout:53: writing smithi07031500-53 from 473964 to 1085030 tid 1 2024-04-16T07:45:57.648 INFO:tasks.rados.rados.0.smithi070.stdout:53: writing smithi07031500-53 from 1826593 to 2372928 tid 2 2024-04-16T07:45:57.652 INFO:tasks.rados.rados.0.smithi070.stdout:53: writing smithi07031500-53 from 2778819 to 2873964 tid 3 2024-04-16T07:45:57.652 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.652 INFO:tasks.rados.rados.0.smithi070.stdout:36: finishing write tid 4 to smithi07031500-36 2024-04-16T07:45:57.652 INFO:tasks.rados.rados.0.smithi070.stdout:36: oid 36 updating version 0 to 8 2024-04-16T07:45:57.652 INFO:tasks.rados.rados.0.smithi070.stdout:36: oid 36 version 8 is already newer than 7 2024-04-16T07:45:57.652 INFO:tasks.rados.rados.0.smithi070.stdout:36: oid 36 updating version 8 to 9 2024-04-16T07:45:57.652 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-16T07:45:57.652 INFO:tasks.rados.rados.0.smithi070.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-04-16T07:45:57.652 INFO:tasks.rados.rados.0.smithi070.stdout:51: finishing write tid 1 to smithi07031500-51 2024-04-16T07:45:57.652 INFO:tasks.rados.rados.0.smithi070.stdout:49: finishing write tid 1 to smithi07031500-49 2024-04-16T07:45:57.652 INFO:tasks.rados.rados.0.smithi070.stdout:44: finishing write tid 2 to smithi07031500-44 2024-04-16T07:45:57.652 INFO:tasks.rados.rados.0.smithi070.stdout:45: finishing write tid 3 to smithi07031500-45 2024-04-16T07:45:57.653 INFO:tasks.rados.rados.0.smithi070.stdout:45: finishing write tid 4 to smithi07031500-45 2024-04-16T07:45:57.653 INFO:tasks.rados.rados.0.smithi070.stdout:45: oid 45 updating version 0 to 8 2024-04-16T07:45:57.653 INFO:tasks.rados.rados.0.smithi070.stdout:45: oid 45 version 8 is already newer than 7 2024-04-16T07:45:57.654 INFO:tasks.rados.rados.0.smithi070.stdout:45: oid 45 updating version 8 to 9 2024-04-16T07:45:57.654 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 45 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-16T07:45:57.654 INFO:tasks.rados.rados.0.smithi070.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-04-16T07:45:57.654 INFO:tasks.rados.rados.0.smithi070.stdout:41: finishing write tid 2 to smithi07031500-41 2024-04-16T07:45:57.654 INFO:tasks.rados.rados.0.smithi070.stdout:36: done (15 left) 2024-04-16T07:45:57.654 INFO:tasks.rados.rados.0.smithi070.stdout:45: done (14 left) 2024-04-16T07:45:57.654 INFO:tasks.rados.rados.0.smithi070.stdout:54: write initial oid 54 2024-04-16T07:45:57.654 INFO:tasks.rados.rados.0.smithi070.stdout:54: seq_num 53 ranges {490771=686315,1885075=406346} 2024-04-16T07:45:57.660 INFO:tasks.rados.rados.0.smithi070.stdout:54: writing smithi07031500-54 from 490771 to 1177086 tid 1 2024-04-16T07:45:57.667 INFO:tasks.rados.rados.0.smithi070.stdout:54: writing smithi07031500-54 from 1885075 to 2291421 tid 2 2024-04-16T07:45:57.667 INFO:tasks.rados.rados.0.smithi070.stdout:43: finishing write tid 3 to smithi07031500-43 2024-04-16T07:45:57.667 INFO:tasks.rados.rados.0.smithi070.stdout:28: finishing write tid 1 to smithi07031500-28 2024-04-16T07:45:57.667 INFO:tasks.rados.rados.0.smithi070.stdout:55: write initial oid 55 2024-04-16T07:45:57.667 INFO:tasks.rados.rados.0.smithi070.stdout:55: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2024-04-16T07:45:57.673 INFO:tasks.rados.rados.0.smithi070.stdout:55: writing smithi07031500-55 from 507578 to 985495 tid 1 2024-04-16T07:45:57.680 INFO:tasks.rados.rados.0.smithi070.stdout:55: writing smithi07031500-55 from 1543557 to 2093561 tid 2 2024-04-16T07:45:57.684 INFO:tasks.rados.rados.0.smithi070.stdout:55: writing smithi07031500-55 from 2833665 to 2907578 tid 3 2024-04-16T07:45:57.684 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.685 INFO:tasks.rados.rados.0.smithi070.stdout:38: finishing write tid 1 to smithi07031500-38 2024-04-16T07:45:57.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:57.683+0000 7f19e139d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:57.686 INFO:tasks.rados.rados.0.smithi070.stdout:43: finishing write tid 4 to smithi07031500-43 2024-04-16T07:45:57.686 INFO:tasks.rados.rados.0.smithi070.stdout:43: oid 43 updating version 0 to 6 2024-04-16T07:45:57.686 INFO:tasks.rados.rados.0.smithi070.stdout:43: oid 43 version 6 is already newer than 4 2024-04-16T07:45:57.686 INFO:tasks.rados.rados.0.smithi070.stdout:43: oid 43 version 6 is already newer than 5 2024-04-16T07:45:57.686 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 43 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-16T07:45:57.687 INFO:tasks.rados.rados.0.smithi070.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-04-16T07:45:57.687 INFO:tasks.rados.rados.0.smithi070.stdout:43: done (15 left) 2024-04-16T07:45:57.687 INFO:tasks.rados.rados.0.smithi070.stdout:56: write initial oid 56 2024-04-16T07:45:57.687 INFO:tasks.rados.rados.0.smithi070.stdout:56: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-04-16T07:45:57.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:45:57.694+0000 7f19e139d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:57.693 INFO:tasks.rados.rados.0.smithi070.stdout:56: writing smithi07031500-56 from 524385 to 1077551 tid 1 2024-04-16T07:45:57.700 INFO:tasks.rados.rados.0.smithi070.stdout:56: writing smithi07031500-56 from 1602039 to 2012054 tid 2 2024-04-16T07:45:57.706 INFO:tasks.rados.rados.0.smithi070.stdout:56: writing smithi07031500-56 from 2577441 to 2924385 tid 3 2024-04-16T07:45:57.706 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.706 INFO:tasks.rados.rados.0.smithi070.stdout:28: finishing write tid 2 to smithi07031500-28 2024-04-16T07:45:57.713 INFO:tasks.rados.rados.0.smithi070.stdout:44: finishing write tid 3 to smithi07031500-44 2024-04-16T07:45:57.715 INFO:tasks.rados.rados.0.smithi070.stdout:28: finishing write tid 3 to smithi07031500-28 2024-04-16T07:45:57.716 INFO:tasks.rados.rados.0.smithi070.stdout:51: finishing write tid 2 to smithi07031500-51 2024-04-16T07:45:57.717 INFO:tasks.rados.rados.0.smithi070.stdout:47: finishing write tid 1 to smithi07031500-47 2024-04-16T07:45:57.718 INFO:tasks.rados.rados.0.smithi070.stdout:44: finishing write tid 4 to smithi07031500-44 2024-04-16T07:45:57.718 INFO:tasks.rados.rados.0.smithi070.stdout:44: oid 44 updating version 0 to 12 2024-04-16T07:45:57.718 INFO:tasks.rados.rados.0.smithi070.stdout:44: oid 44 version 12 is already newer than 10 2024-04-16T07:45:57.718 INFO:tasks.rados.rados.0.smithi070.stdout:44: oid 44 version 12 is already newer than 11 2024-04-16T07:45:57.718 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 44 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-16T07:45:57.718 INFO:tasks.rados.rados.0.smithi070.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-04-16T07:45:57.718 INFO:tasks.rados.rados.0.smithi070.stdout:44: done (15 left) 2024-04-16T07:45:57.718 INFO:tasks.rados.rados.0.smithi070.stdout:57: write initial oid 57 2024-04-16T07:45:57.719 INFO:tasks.rados.rados.0.smithi070.stdout:57: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2024-04-16T07:45:57.725 INFO:tasks.rados.rados.0.smithi070.stdout:57: writing smithi07031500-57 from 541192 to 1169607 tid 1 2024-04-16T07:45:57.731 INFO:tasks.rados.rados.0.smithi070.stdout:57: writing smithi07031500-57 from 1660521 to 2214194 tid 2 2024-04-16T07:45:57.738 INFO:tasks.rados.rados.0.smithi070.stdout:57: writing smithi07031500-57 from 2888511 to 2941192 tid 3 2024-04-16T07:45:57.738 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.738 INFO:tasks.rados.rados.0.smithi070.stdout:28: finishing write tid 4 to smithi07031500-28 2024-04-16T07:45:57.738 INFO:tasks.rados.rados.0.smithi070.stdout:28: oid 28 updating version 0 to 16 2024-04-16T07:45:57.738 INFO:tasks.rados.rados.0.smithi070.stdout:28: oid 28 updating version 16 to 18 2024-04-16T07:45:57.738 INFO:tasks.rados.rados.0.smithi070.stdout:28: oid 28 version 18 is already newer than 18 2024-04-16T07:45:57.738 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 28 v 18 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-16T07:45:57.738 INFO:tasks.rados.rados.0.smithi070.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-04-16T07:45:57.738 INFO:tasks.rados.rados.0.smithi070.stdout:28: done (15 left) 2024-04-16T07:45:57.739 INFO:tasks.rados.rados.0.smithi070.stdout:58: write initial oid 58 2024-04-16T07:45:57.739 INFO:tasks.rados.rados.0.smithi070.stdout:58: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2024-04-16T07:45:57.745 INFO:tasks.rados.rados.0.smithi070.stdout:58: writing smithi07031500-58 from 557999 to 1261663 tid 1 2024-04-16T07:45:57.751 INFO:tasks.rados.rados.0.smithi070.stdout:58: writing smithi07031500-58 from 1719003 to 2132687 tid 2 2024-04-16T07:45:57.756 INFO:tasks.rados.rados.0.smithi070.stdout:58: writing smithi07031500-58 from 2632287 to 2957999 tid 3 2024-04-16T07:45:57.756 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.757 INFO:tasks.rados.rados.0.smithi070.stdout:53: finishing write tid 1 to smithi07031500-53 2024-04-16T07:45:57.757 INFO:tasks.rados.rados.0.smithi070.stdout:49: finishing write tid 2 to smithi07031500-49 2024-04-16T07:45:57.757 INFO:tasks.rados.rados.0.smithi070.stdout:49: finishing write tid 3 to smithi07031500-49 2024-04-16T07:45:57.757 INFO:tasks.rados.rados.0.smithi070.stdout:49: finishing write tid 4 to smithi07031500-49 2024-04-16T07:45:57.757 INFO:tasks.rados.rados.0.smithi070.stdout:49: oid 49 updating version 0 to 12 2024-04-16T07:45:57.757 INFO:tasks.rados.rados.0.smithi070.stdout:49: oid 49 version 12 is already newer than 12 2024-04-16T07:45:57.757 INFO:tasks.rados.rados.0.smithi070.stdout:49: oid 49 version 12 is already newer than 10 2024-04-16T07:45:57.757 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-16T07:45:57.757 INFO:tasks.rados.rados.0.smithi070.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-04-16T07:45:57.757 INFO:tasks.rados.rados.0.smithi070.stdout:41: finishing write tid 3 to smithi07031500-41 2024-04-16T07:45:57.757 INFO:tasks.rados.rados.0.smithi070.stdout:41: finishing write tid 4 to smithi07031500-41 2024-04-16T07:45:57.757 INFO:tasks.rados.rados.0.smithi070.stdout:41: oid 41 updating version 0 to 8 2024-04-16T07:45:57.757 INFO:tasks.rados.rados.0.smithi070.stdout:41: oid 41 updating version 8 to 9 2024-04-16T07:45:57.757 INFO:tasks.rados.rados.0.smithi070.stdout:41: oid 41 version 9 is already newer than 7 2024-04-16T07:45:57.757 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 41 v 9 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-16T07:45:57.758 INFO:tasks.rados.rados.0.smithi070.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-04-16T07:45:57.758 INFO:tasks.rados.rados.0.smithi070.stdout:41: done (15 left) 2024-04-16T07:45:57.758 INFO:tasks.rados.rados.0.smithi070.stdout:49: done (14 left) 2024-04-16T07:45:57.758 INFO:tasks.rados.rados.0.smithi070.stdout:59: write initial oid 59 2024-04-16T07:45:57.758 INFO:tasks.rados.rados.0.smithi070.stdout:59: seq_num 58 ranges {574806=778913,2061132=557342} 2024-04-16T07:45:57.764 INFO:tasks.rados.rados.0.smithi070.stdout:59: writing smithi07031500-59 from 574806 to 1353719 tid 1 2024-04-16T07:45:57.774 INFO:tasks.rados.rados.0.smithi070.stdout:59: writing smithi07031500-59 from 2061132 to 2618474 tid 2 2024-04-16T07:45:57.774 INFO:tasks.rados.rados.0.smithi070.stdout:60: write initial oid 60 2024-04-16T07:45:57.774 INFO:tasks.rados.rados.0.smithi070.stdout:60: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2024-04-16T07:45:57.782 INFO:tasks.rados.rados.0.smithi070.stdout:60: writing smithi07031500-60 from 591613 to 1045775 tid 1 2024-04-16T07:45:57.791 INFO:tasks.rados.rados.0.smithi070.stdout:60: writing smithi07031500-60 from 1719614 to 2136967 tid 2 2024-04-16T07:45:57.797 INFO:tasks.rados.rados.0.smithi070.stdout:60: writing smithi07031500-60 from 2570780 to 2991613 tid 3 2024-04-16T07:45:57.797 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.797 INFO:tasks.rados.rados.0.smithi070.stdout:54: finishing write tid 1 to smithi07031500-54 2024-04-16T07:45:57.797 INFO:tasks.rados.rados.0.smithi070.stdout:38: finishing write tid 2 to smithi07031500-38 2024-04-16T07:45:57.797 INFO:tasks.rados.rados.0.smithi070.stdout:51: finishing write tid 3 to smithi07031500-51 2024-04-16T07:45:57.797 INFO:tasks.rados.rados.0.smithi070.stdout:46: finishing write tid 1 to smithi07031500-46 2024-04-16T07:45:57.798 INFO:tasks.rados.rados.0.smithi070.stdout:51: finishing write tid 4 to smithi07031500-51 2024-04-16T07:45:57.798 INFO:tasks.rados.rados.0.smithi070.stdout:53: finishing write tid 2 to smithi07031500-53 2024-04-16T07:45:57.802 INFO:tasks.rados.rados.0.smithi070.stdout:51: finishing write tid 5 to smithi07031500-51 2024-04-16T07:45:57.802 INFO:tasks.rados.rados.0.smithi070.stdout:51: oid 51 updating version 0 to 19 2024-04-16T07:45:57.802 INFO:tasks.rados.rados.0.smithi070.stdout:51: oid 51 version 19 is already newer than 19 2024-04-16T07:45:57.802 INFO:tasks.rados.rados.0.smithi070.stdout:51: oid 51 version 19 is already newer than 16 2024-04-16T07:45:57.802 INFO:tasks.rados.rados.0.smithi070.stdout:51: oid 51 version 19 is already newer than 17 2024-04-16T07:45:57.802 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 51 v 19 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-04-16T07:45:57.803 INFO:tasks.rados.rados.0.smithi070.stdout:51: left oid 51 (ObjNum 50 snap 0 seq_num 50) 2024-04-16T07:45:57.803 INFO:tasks.rados.rados.0.smithi070.stdout:51: done (15 left) 2024-04-16T07:45:57.803 INFO:tasks.rados.rados.0.smithi070.stdout:61: write initial oid 61 2024-04-16T07:45:57.803 INFO:tasks.rados.rados.0.smithi070.stdout:61: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-04-16T07:45:57.808 INFO:tasks.rados.rados.0.smithi070.stdout:61: writing smithi07031500-61 from 608420 to 1137831 tid 1 2024-04-16T07:45:57.816 INFO:tasks.rados.rados.0.smithi070.stdout:61: writing smithi07031500-61 from 1778096 to 2339107 tid 2 2024-04-16T07:45:57.820 INFO:tasks.rados.rados.0.smithi070.stdout:61: writing smithi07031500-61 from 2881850 to 3008420 tid 3 2024-04-16T07:45:57.820 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.821 INFO:tasks.rados.rados.0.smithi070.stdout:47: finishing write tid 2 to smithi07031500-47 2024-04-16T07:45:57.821 INFO:tasks.rados.rados.0.smithi070.stdout:47: finishing write tid 3 to smithi07031500-47 2024-04-16T07:45:57.821 INFO:tasks.rados.rados.0.smithi070.stdout:54: finishing write tid 2 to smithi07031500-54 2024-04-16T07:45:57.821 INFO:tasks.rados.rados.0.smithi070.stdout:47: finishing write tid 4 to smithi07031500-47 2024-04-16T07:45:57.821 INFO:tasks.rados.rados.0.smithi070.stdout:47: oid 47 updating version 0 to 12 2024-04-16T07:45:57.821 INFO:tasks.rados.rados.0.smithi070.stdout:47: oid 47 version 12 is already newer than 10 2024-04-16T07:45:57.821 INFO:tasks.rados.rados.0.smithi070.stdout:47: oid 47 version 12 is already newer than 12 2024-04-16T07:45:57.821 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-16T07:45:57.821 INFO:tasks.rados.rados.0.smithi070.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-04-16T07:45:57.821 INFO:tasks.rados.rados.0.smithi070.stdout:47: done (15 left) 2024-04-16T07:45:57.821 INFO:tasks.rados.rados.0.smithi070.stdout:62: write initial oid 62 2024-04-16T07:45:57.821 INFO:tasks.rados.rados.0.smithi070.stdout:62: seq_num 61 ranges {625227=721013,1952931=704669} 2024-04-16T07:45:57.830 INFO:tasks.rados.rados.0.smithi070.stdout:62: writing smithi07031500-62 from 625227 to 1346240 tid 1 2024-04-16T07:45:57.840 INFO:tasks.rados.rados.0.smithi070.stdout:62: writing smithi07031500-62 from 1952931 to 2657600 tid 2 2024-04-16T07:45:57.840 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.840 INFO:tasks.rados.rados.0.smithi070.stdout:58: finishing write tid 1 to smithi07031500-58 2024-04-16T07:45:57.840 INFO:tasks.rados.rados.0.smithi070.stdout:59: finishing write tid 1 to smithi07031500-59 2024-04-16T07:45:57.840 INFO:tasks.rados.rados.0.smithi070.stdout:50: finishing write tid 1 to smithi07031500-50 2024-04-16T07:45:57.840 INFO:tasks.rados.rados.0.smithi070.stdout:46: finishing write tid 2 to smithi07031500-46 2024-04-16T07:45:57.840 INFO:tasks.rados.rados.0.smithi070.stdout:53: finishing write tid 3 to smithi07031500-53 2024-04-16T07:45:57.840 INFO:tasks.rados.rados.0.smithi070.stdout:46: finishing write tid 3 to smithi07031500-46 2024-04-16T07:45:57.841 INFO:tasks.rados.rados.0.smithi070.stdout:38: finishing write tid 3 to smithi07031500-38 2024-04-16T07:45:57.847 INFO:tasks.rados.rados.0.smithi070.stdout:38: finishing write tid 4 to smithi07031500-38 2024-04-16T07:45:57.847 INFO:tasks.rados.rados.0.smithi070.stdout:38: oid 38 updating version 0 to 12 2024-04-16T07:45:57.847 INFO:tasks.rados.rados.0.smithi070.stdout:38: oid 38 version 12 is already newer than 10 2024-04-16T07:45:57.847 INFO:tasks.rados.rados.0.smithi070.stdout:38: oid 38 version 12 is already newer than 11 2024-04-16T07:45:57.847 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 38 v 12 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-16T07:45:57.848 INFO:tasks.rados.rados.0.smithi070.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-04-16T07:45:57.848 INFO:tasks.rados.rados.0.smithi070.stdout:38: done (15 left) 2024-04-16T07:45:57.848 INFO:tasks.rados.rados.0.smithi070.stdout:63: write initial oid 63 2024-04-16T07:45:57.848 INFO:tasks.rados.rados.0.smithi070.stdout:63: seq_num 62 ranges {642034=796262,1895060=564680} 2024-04-16T07:45:57.855 INFO:tasks.rados.rados.0.smithi070.stdout:63: writing smithi07031500-63 from 642034 to 1438296 tid 1 2024-04-16T07:45:57.862 INFO:tasks.rados.rados.0.smithi070.stdout:63: writing smithi07031500-63 from 1895060 to 2459740 tid 2 2024-04-16T07:45:57.862 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.862 INFO:tasks.rados.rados.0.smithi070.stdout:54: finishing write tid 3 to smithi07031500-54 2024-04-16T07:45:57.862 INFO:tasks.rados.rados.0.smithi070.stdout:46: finishing write tid 4 to smithi07031500-46 2024-04-16T07:45:57.862 INFO:tasks.rados.rados.0.smithi070.stdout:46: oid 46 updating version 0 to 21 2024-04-16T07:45:57.862 INFO:tasks.rados.rados.0.smithi070.stdout:46: oid 46 version 21 is already newer than 21 2024-04-16T07:45:57.862 INFO:tasks.rados.rados.0.smithi070.stdout:46: oid 46 version 21 is already newer than 19 2024-04-16T07:45:57.862 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 46 v 21 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-16T07:45:57.862 INFO:tasks.rados.rados.0.smithi070.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-04-16T07:45:57.863 INFO:tasks.rados.rados.0.smithi070.stdout:54: finishing write tid 4 to smithi07031500-54 2024-04-16T07:45:57.863 INFO:tasks.rados.rados.0.smithi070.stdout:54: oid 54 updating version 0 to 10 2024-04-16T07:45:57.863 INFO:tasks.rados.rados.0.smithi070.stdout:54: oid 54 updating version 10 to 11 2024-04-16T07:45:57.863 INFO:tasks.rados.rados.0.smithi070.stdout:54: oid 54 updating version 11 to 12 2024-04-16T07:45:57.863 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 54 v 12 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-04-16T07:45:57.863 INFO:tasks.rados.rados.0.smithi070.stdout:54: left oid 54 (ObjNum 53 snap 0 seq_num 53) 2024-04-16T07:45:57.863 INFO:tasks.rados.rados.0.smithi070.stdout:46: done (15 left) 2024-04-16T07:45:57.863 INFO:tasks.rados.rados.0.smithi070.stdout:54: done (14 left) 2024-04-16T07:45:57.863 INFO:tasks.rados.rados.0.smithi070.stdout:64: write initial oid 64 2024-04-16T07:45:57.863 INFO:tasks.rados.rados.0.smithi070.stdout:64: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-04-16T07:45:57.869 INFO:tasks.rados.rados.0.smithi070.stdout:64: writing smithi07031500-64 from 658841 to 1130352 tid 1 2024-04-16T07:45:57.879 INFO:tasks.rados.rados.0.smithi070.stdout:64: writing smithi07031500-64 from 1553542 to 2261880 tid 2 2024-04-16T07:45:57.884 INFO:tasks.rados.rados.0.smithi070.stdout:64: writing smithi07031500-64 from 2964119 to 3058841 tid 3 2024-04-16T07:45:57.884 INFO:tasks.rados.rados.0.smithi070.stdout:65: write initial oid 65 2024-04-16T07:45:57.884 INFO:tasks.rados.rados.0.smithi070.stdout:65: seq_num 64 ranges {675648=546760,2012024=568349} 2024-04-16T07:45:57.892 INFO:tasks.rados.rados.0.smithi070.stdout:65: writing smithi07031500-65 from 675648 to 1222408 tid 1 2024-04-16T07:45:57.903 INFO:tasks.rados.rados.0.smithi070.stdout:65: writing smithi07031500-65 from 2012024 to 2580373 tid 2 2024-04-16T07:45:57.903 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.903 INFO:tasks.rados.rados.0.smithi070.stdout:59: finishing write tid 2 to smithi07031500-59 2024-04-16T07:45:57.903 INFO:tasks.rados.rados.0.smithi070.stdout:52: finishing write tid 1 to smithi07031500-52 2024-04-16T07:45:57.903 INFO:tasks.rados.rados.0.smithi070.stdout:50: finishing write tid 2 to smithi07031500-50 2024-04-16T07:45:57.907 INFO:tasks.rados.rados.0.smithi070.stdout:59: finishing write tid 3 to smithi07031500-59 2024-04-16T07:45:57.910 INFO:tasks.rados.rados.0.smithi070.stdout:53: finishing write tid 4 to smithi07031500-53 2024-04-16T07:45:57.914 INFO:tasks.rados.rados.0.smithi070.stdout:58: finishing write tid 2 to smithi07031500-58 2024-04-16T07:45:57.915 INFO:tasks.rados.rados.0.smithi070.stdout:53: finishing write tid 5 to smithi07031500-53 2024-04-16T07:45:57.915 INFO:tasks.rados.rados.0.smithi070.stdout:53: oid 53 updating version 0 to 10 2024-04-16T07:45:57.915 INFO:tasks.rados.rados.0.smithi070.stdout:53: oid 53 version 10 is already newer than 8 2024-04-16T07:45:57.915 INFO:tasks.rados.rados.0.smithi070.stdout:53: oid 53 version 10 is already newer than 7 2024-04-16T07:45:57.915 INFO:tasks.rados.rados.0.smithi070.stdout:53: oid 53 version 10 is already newer than 9 2024-04-16T07:45:57.915 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 53 v 10 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-04-16T07:45:57.915 INFO:tasks.rados.rados.0.smithi070.stdout:53: left oid 53 (ObjNum 52 snap 0 seq_num 52) 2024-04-16T07:45:57.915 INFO:tasks.rados.rados.0.smithi070.stdout:53: done (15 left) 2024-04-16T07:45:57.915 INFO:tasks.rados.rados.0.smithi070.stdout:66: write initial oid 66 2024-04-16T07:45:57.915 INFO:tasks.rados.rados.0.smithi070.stdout:66: seq_num 65 ranges {692455=622009,2070506=712007} 2024-04-16T07:45:57.923 INFO:tasks.rados.rados.0.smithi070.stdout:66: writing smithi07031500-66 from 692455 to 1314464 tid 1 2024-04-16T07:45:57.933 INFO:tasks.rados.rados.0.smithi070.stdout:66: writing smithi07031500-66 from 2070506 to 2782513 tid 2 2024-04-16T07:45:57.933 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.933 INFO:tasks.rados.rados.0.smithi070.stdout:48: finishing write tid 1 to smithi07031500-48 2024-04-16T07:45:57.934 INFO:tasks.rados.rados.0.smithi070.stdout:39: finishing write tid 1 to smithi07031500-39 2024-04-16T07:45:57.934 INFO:tasks.rados.rados.0.smithi070.stdout:50: finishing write tid 3 to smithi07031500-50 2024-04-16T07:45:57.934 INFO:tasks.rados.rados.0.smithi070.stdout:59: finishing write tid 4 to smithi07031500-59 2024-04-16T07:45:57.934 INFO:tasks.rados.rados.0.smithi070.stdout:59: oid 59 updating version 0 to 4 2024-04-16T07:45:57.934 INFO:tasks.rados.rados.0.smithi070.stdout:59: oid 59 updating version 4 to 5 2024-04-16T07:45:57.934 INFO:tasks.rados.rados.0.smithi070.stdout:59: oid 59 updating version 5 to 6 2024-04-16T07:45:57.934 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 59 v 6 (ObjNum 58 snap 0 seq_num 58) dirty exists 2024-04-16T07:45:57.934 INFO:tasks.rados.rados.0.smithi070.stdout:59: left oid 59 (ObjNum 58 snap 0 seq_num 58) 2024-04-16T07:45:57.934 INFO:tasks.rados.rados.0.smithi070.stdout:55: finishing write tid 1 to smithi07031500-55 2024-04-16T07:45:57.934 INFO:tasks.rados.rados.0.smithi070.stdout:50: finishing write tid 4 to smithi07031500-50 2024-04-16T07:45:57.934 INFO:tasks.rados.rados.0.smithi070.stdout:50: oid 50 updating version 0 to 12 2024-04-16T07:45:57.934 INFO:tasks.rados.rados.0.smithi070.stdout:50: oid 50 version 12 is already newer than 10 2024-04-16T07:45:57.935 INFO:tasks.rados.rados.0.smithi070.stdout:50: oid 50 version 12 is already newer than 12 2024-04-16T07:45:57.935 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 50 v 12 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-16T07:45:57.935 INFO:tasks.rados.rados.0.smithi070.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-04-16T07:45:57.935 INFO:tasks.rados.rados.0.smithi070.stdout:50: done (15 left) 2024-04-16T07:45:57.935 INFO:tasks.rados.rados.0.smithi070.stdout:59: done (14 left) 2024-04-16T07:45:57.935 INFO:tasks.rados.rados.0.smithi070.stdout:67: write initial oid 67 2024-04-16T07:45:57.935 INFO:tasks.rados.rados.0.smithi070.stdout:67: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2024-04-16T07:45:57.944 INFO:tasks.rados.rados.0.smithi070.stdout:67: writing smithi07031500-67 from 709262 to 1406520 tid 1 2024-04-16T07:45:57.950 INFO:tasks.rados.rados.0.smithi070.stdout:67: writing smithi07031500-67 from 2012635 to 2584653 tid 2 2024-04-16T07:45:57.954 INFO:tasks.rados.rados.0.smithi070.stdout:67: writing smithi07031500-67 from 3046388 to 3109262 tid 3 2024-04-16T07:45:57.954 INFO:tasks.rados.rados.0.smithi070.stdout:68: write initial oid 68 2024-04-16T07:45:57.954 INFO:tasks.rados.rados.0.smithi070.stdout:68: seq_num 67 ranges {726069=772507,2071117=715676} 2024-04-16T07:45:57.964 INFO:tasks.rados.rados.0.smithi070.stdout:68: writing smithi07031500-68 from 726069 to 1498576 tid 1 2024-04-16T07:45:57.971 INFO:tasks.rados.rados.0.smithi070.stdout:68: writing smithi07031500-68 from 2071117 to 2786793 tid 2 2024-04-16T07:45:57.972 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:57.972 INFO:tasks.rados.rados.0.smithi070.stdout:65: finishing write tid 1 to smithi07031500-65 2024-04-16T07:45:57.972 INFO:tasks.rados.rados.0.smithi070.stdout:52: finishing write tid 2 to smithi07031500-52 2024-04-16T07:45:57.972 INFO:tasks.rados.rados.0.smithi070.stdout:48: finishing write tid 2 to smithi07031500-48 2024-04-16T07:45:57.978 INFO:tasks.rados.rados.0.smithi070.stdout:48: finishing write tid 3 to smithi07031500-48 2024-04-16T07:45:57.981 INFO:tasks.rados.rados.0.smithi070.stdout:48: finishing write tid 4 to smithi07031500-48 2024-04-16T07:45:57.981 INFO:tasks.rados.rados.0.smithi070.stdout:48: oid 48 updating version 0 to 24 2024-04-16T07:45:57.981 INFO:tasks.rados.rados.0.smithi070.stdout:48: oid 48 version 24 is already newer than 24 2024-04-16T07:45:57.981 INFO:tasks.rados.rados.0.smithi070.stdout:48: oid 48 version 24 is already newer than 22 2024-04-16T07:45:57.981 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 48 v 24 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-16T07:45:57.982 INFO:tasks.rados.rados.0.smithi070.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-04-16T07:45:57.982 INFO:tasks.rados.rados.0.smithi070.stdout:48: done (15 left) 2024-04-16T07:45:57.982 INFO:tasks.rados.rados.0.smithi070.stdout:69: write initial oid 69 2024-04-16T07:45:57.982 INFO:tasks.rados.rados.0.smithi070.stdout:69: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2024-04-16T07:45:57.990 INFO:tasks.rados.rados.0.smithi070.stdout:69: writing smithi07031500-69 from 742876 to 1190632 tid 1 2024-04-16T07:45:58.000 INFO:tasks.rados.rados.0.smithi070.stdout:69: writing smithi07031500-69 from 1729599 to 2305286 tid 2 2024-04-16T07:45:58.006 INFO:tasks.rados.rados.0.smithi070.stdout:69: writing smithi07031500-69 from 3101234 to 3142876 tid 3 2024-04-16T07:45:58.006 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:58.006 INFO:tasks.rados.rados.0.smithi070.stdout:63: finishing write tid 1 to smithi07031500-63 2024-04-16T07:45:58.006 INFO:tasks.rados.rados.0.smithi070.stdout:58: finishing write tid 3 to smithi07031500-58 2024-04-16T07:45:58.006 INFO:tasks.rados.rados.0.smithi070.stdout:39: finishing write tid 2 to smithi07031500-39 2024-04-16T07:45:58.006 INFO:tasks.rados.rados.0.smithi070.stdout:62: finishing write tid 1 to smithi07031500-62 2024-04-16T07:45:58.006 INFO:tasks.rados.rados.0.smithi070.stdout:52: finishing write tid 3 to smithi07031500-52 2024-04-16T07:45:58.006 INFO:tasks.rados.rados.0.smithi070.stdout:52: finishing write tid 4 to smithi07031500-52 2024-04-16T07:45:58.007 INFO:tasks.rados.rados.0.smithi070.stdout:52: oid 52 updating version 0 to 13 2024-04-16T07:45:58.007 INFO:tasks.rados.rados.0.smithi070.stdout:52: oid 52 updating version 13 to 15 2024-04-16T07:45:58.007 INFO:tasks.rados.rados.0.smithi070.stdout:52: oid 52 version 15 is already newer than 15 2024-04-16T07:45:58.007 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 52 v 15 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-04-16T07:45:58.007 INFO:tasks.rados.rados.0.smithi070.stdout:52: left oid 52 (ObjNum 51 snap 0 seq_num 51) 2024-04-16T07:45:58.007 INFO:tasks.rados.rados.0.smithi070.stdout:39: finishing write tid 3 to smithi07031500-39 2024-04-16T07:45:58.007 INFO:tasks.rados.rados.0.smithi070.stdout:61: finishing write tid 1 to smithi07031500-61 2024-04-16T07:45:58.007 INFO:tasks.rados.rados.0.smithi070.stdout:52: done (15 left) 2024-04-16T07:45:58.007 INFO:tasks.rados.rados.0.smithi070.stdout:70: write initial oid 70 2024-04-16T07:45:58.007 INFO:tasks.rados.rados.0.smithi070.stdout:70: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2024-04-16T07:45:58.015 INFO:tasks.rados.rados.0.smithi070.stdout:70: writing smithi07031500-70 from 759683 to 1399041 tid 1 2024-04-16T07:45:58.022 INFO:tasks.rados.rados.0.smithi070.stdout:70: writing smithi07031500-70 from 1904434 to 2623779 tid 2 2024-04-16T07:45:58.026 INFO:tasks.rados.rados.0.smithi070.stdout:70: writing smithi07031500-70 from 3128657 to 3159683 tid 3 2024-04-16T07:45:58.026 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:58.026 INFO:tasks.rados.rados.0.smithi070.stdout:39: finishing write tid 4 to smithi07031500-39 2024-04-16T07:45:58.026 INFO:tasks.rados.rados.0.smithi070.stdout:39: oid 39 updating version 0 to 15 2024-04-16T07:45:58.026 INFO:tasks.rados.rados.0.smithi070.stdout:39: oid 39 version 15 is already newer than 13 2024-04-16T07:45:58.026 INFO:tasks.rados.rados.0.smithi070.stdout:39: oid 39 version 15 is already newer than 15 2024-04-16T07:45:58.027 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 39 v 15 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-16T07:45:58.027 INFO:tasks.rados.rados.0.smithi070.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-04-16T07:45:58.027 INFO:tasks.rados.rados.0.smithi070.stdout:55: finishing write tid 2 to smithi07031500-55 2024-04-16T07:45:58.027 INFO:tasks.rados.rados.0.smithi070.stdout:55: finishing write tid 3 to smithi07031500-55 2024-04-16T07:45:58.027 INFO:tasks.rados.rados.0.smithi070.stdout:65: finishing write tid 2 to smithi07031500-65 2024-04-16T07:45:58.027 INFO:tasks.rados.rados.0.smithi070.stdout:39: done (15 left) 2024-04-16T07:45:58.027 INFO:tasks.rados.rados.0.smithi070.stdout:71: write initial oid 71 2024-04-16T07:45:58.027 INFO:tasks.rados.rados.0.smithi070.stdout:71: seq_num 70 ranges {776490=714607,2246563=579356} 2024-04-16T07:45:58.037 INFO:tasks.rados.rados.0.smithi070.stdout:71: writing smithi07031500-71 from 776490 to 1491097 tid 1 2024-04-16T07:45:58.048 INFO:tasks.rados.rados.0.smithi070.stdout:71: writing smithi07031500-71 from 2246563 to 2825919 tid 2 2024-04-16T07:45:58.048 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:58.048 INFO:tasks.rados.rados.0.smithi070.stdout:57: finishing write tid 1 to smithi07031500-57 2024-04-16T07:45:58.048 INFO:tasks.rados.rados.0.smithi070.stdout:55: finishing write tid 4 to smithi07031500-55 2024-04-16T07:45:58.048 INFO:tasks.rados.rados.0.smithi070.stdout:62: finishing write tid 2 to smithi07031500-62 2024-04-16T07:45:58.049 INFO:tasks.rados.rados.0.smithi070.stdout:58: finishing write tid 4 to smithi07031500-58 2024-04-16T07:45:58.049 INFO:tasks.rados.rados.0.smithi070.stdout:58: finishing write tid 5 to smithi07031500-58 2024-04-16T07:45:58.049 INFO:tasks.rados.rados.0.smithi070.stdout:58: oid 58 updating version 0 to 14 2024-04-16T07:45:58.049 INFO:tasks.rados.rados.0.smithi070.stdout:58: oid 58 updating version 14 to 16 2024-04-16T07:45:58.049 INFO:tasks.rados.rados.0.smithi070.stdout:58: oid 58 version 16 is already newer than 15 2024-04-16T07:45:58.049 INFO:tasks.rados.rados.0.smithi070.stdout:58: oid 58 version 16 is already newer than 13 2024-04-16T07:45:58.049 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 58 v 16 (ObjNum 57 snap 0 seq_num 57) dirty exists 2024-04-16T07:45:58.049 INFO:tasks.rados.rados.0.smithi070.stdout:58: left oid 58 (ObjNum 57 snap 0 seq_num 57) 2024-04-16T07:45:58.049 INFO:tasks.rados.rados.0.smithi070.stdout:62: finishing write tid 3 to smithi07031500-62 2024-04-16T07:45:58.049 INFO:tasks.rados.rados.0.smithi070.stdout:58: done (15 left) 2024-04-16T07:45:58.049 INFO:tasks.rados.rados.0.smithi070.stdout:72: write initial oid 72 2024-04-16T07:45:58.049 INFO:tasks.rados.rados.0.smithi070.stdout:72: seq_num 71 ranges {793297=789856,2305045=723014} 2024-04-16T07:45:58.060 INFO:tasks.rados.rados.0.smithi070.stdout:72: writing smithi07031500-72 from 793297 to 1583153 tid 1 2024-04-16T07:45:58.072 INFO:tasks.rados.rados.0.smithi070.stdout:72: writing smithi07031500-72 from 2305045 to 3028059 tid 2 2024-04-16T07:45:58.073 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:58.073 INFO:tasks.rados.rados.0.smithi070.stdout:55: finishing write tid 5 to smithi07031500-55 2024-04-16T07:45:58.073 INFO:tasks.rados.rados.0.smithi070.stdout:55: oid 55 updating version 0 to 13 2024-04-16T07:45:58.073 INFO:tasks.rados.rados.0.smithi070.stdout:55: oid 55 updating version 13 to 16 2024-04-16T07:45:58.073 INFO:tasks.rados.rados.0.smithi070.stdout:55: oid 55 version 16 is already newer than 16 2024-04-16T07:45:58.073 INFO:tasks.rados.rados.0.smithi070.stdout:55: oid 55 version 16 is already newer than 16 2024-04-16T07:45:58.073 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 55 v 16 (ObjNum 54 snap 0 seq_num 54) dirty exists 2024-04-16T07:45:58.073 INFO:tasks.rados.rados.0.smithi070.stdout:55: left oid 55 (ObjNum 54 snap 0 seq_num 54) 2024-04-16T07:45:58.073 INFO:tasks.rados.rados.0.smithi070.stdout:62: finishing write tid 4 to smithi07031500-62 2024-04-16T07:45:58.073 INFO:tasks.rados.rados.0.smithi070.stdout:62: oid 62 updating version 0 to 9 2024-04-16T07:45:58.073 INFO:tasks.rados.rados.0.smithi070.stdout:62: oid 62 version 9 is already newer than 7 2024-04-16T07:45:58.073 INFO:tasks.rados.rados.0.smithi070.stdout:62: oid 62 version 9 is already newer than 9 2024-04-16T07:45:58.073 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 62 v 9 (ObjNum 61 snap 0 seq_num 61) dirty exists 2024-04-16T07:45:58.073 INFO:tasks.rados.rados.0.smithi070.stdout:62: left oid 62 (ObjNum 61 snap 0 seq_num 61) 2024-04-16T07:45:58.073 INFO:tasks.rados.rados.0.smithi070.stdout:63: finishing write tid 2 to smithi07031500-63 2024-04-16T07:45:58.073 INFO:tasks.rados.rados.0.smithi070.stdout:56: finishing write tid 1 to smithi07031500-56 2024-04-16T07:45:58.074 INFO:tasks.rados.rados.0.smithi070.stdout:57: finishing write tid 2 to smithi07031500-57 2024-04-16T07:45:58.074 INFO:tasks.rados.rados.0.smithi070.stdout:61: finishing write tid 2 to smithi07031500-61 2024-04-16T07:45:58.074 INFO:tasks.rados.rados.0.smithi070.stdout:65: finishing write tid 3 to smithi07031500-65 2024-04-16T07:45:58.074 INFO:tasks.rados.rados.0.smithi070.stdout:55: done (15 left) 2024-04-16T07:45:58.074 INFO:tasks.rados.rados.0.smithi070.stdout:62: done (14 left) 2024-04-16T07:45:58.074 INFO:tasks.rados.rados.0.smithi070.stdout:73: write initial oid 73 2024-04-16T07:45:58.074 INFO:tasks.rados.rados.0.smithi070.stdout:73: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2024-04-16T07:45:58.079 INFO:tasks.rados.rados.0.smithi070.stdout:73: writing smithi07031500-73 from 410104 to 875209 tid 1 2024-04-16T07:45:58.087 INFO:tasks.rados.rados.0.smithi070.stdout:73: writing smithi07031500-73 from 1563527 to 2146552 tid 2 2024-04-16T07:45:58.094 INFO:tasks.rados.rados.0.smithi070.stdout:73: writing smithi07031500-73 from 2810926 to 3210104 tid 3 2024-04-16T07:45:58.094 INFO:tasks.rados.rados.0.smithi070.stdout:74: write initial oid 74 2024-04-16T07:45:58.094 INFO:tasks.rados.rados.0.smithi070.stdout:74: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2024-04-16T07:45:58.100 INFO:tasks.rados.rados.0.smithi070.stdout:74: writing smithi07031500-74 from 426911 to 967265 tid 1 2024-04-16T07:45:58.111 INFO:tasks.rados.rados.0.smithi070.stdout:74: writing smithi07031500-74 from 1622009 to 2348692 tid 2 2024-04-16T07:45:58.117 INFO:tasks.rados.rados.0.smithi070.stdout:74: writing smithi07031500-74 from 3121996 to 3226911 tid 3 2024-04-16T07:45:58.117 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:58.117 INFO:tasks.rados.rados.0.smithi070.stdout:65: finishing write tid 4 to smithi07031500-65 2024-04-16T07:45:58.117 INFO:tasks.rados.rados.0.smithi070.stdout:65: oid 65 updating version 0 to 7 2024-04-16T07:45:58.117 INFO:tasks.rados.rados.0.smithi070.stdout:65: oid 65 updating version 7 to 9 2024-04-16T07:45:58.117 INFO:tasks.rados.rados.0.smithi070.stdout:65: oid 65 version 9 is already newer than 8 2024-04-16T07:45:58.117 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 65 v 9 (ObjNum 64 snap 0 seq_num 64) dirty exists 2024-04-16T07:45:58.117 INFO:tasks.rados.rados.0.smithi070.stdout:65: left oid 65 (ObjNum 64 snap 0 seq_num 64) 2024-04-16T07:45:58.117 INFO:tasks.rados.rados.0.smithi070.stdout:65: done (15 left) 2024-04-16T07:45:58.117 INFO:tasks.rados.rados.0.smithi070.stdout:75: write initial oid 75 2024-04-16T07:45:58.118 INFO:tasks.rados.rados.0.smithi070.stdout:75: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2024-04-16T07:45:58.123 INFO:tasks.rados.rados.0.smithi070.stdout:75: writing smithi07031500-75 from 443718 to 1059321 tid 1 2024-04-16T07:45:58.129 INFO:tasks.rados.rados.0.smithi070.stdout:75: writing smithi07031500-75 from 1564138 to 2034479 tid 2 2024-04-16T07:45:58.137 INFO:tasks.rados.rados.0.smithi070.stdout:75: writing smithi07031500-75 from 2633066 to 3243718 tid 3 2024-04-16T07:45:58.138 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:58.138 INFO:tasks.rados.rados.0.smithi070.stdout:61: finishing write tid 3 to smithi07031500-61 2024-04-16T07:45:58.138 INFO:tasks.rados.rados.0.smithi070.stdout:57: finishing write tid 3 to smithi07031500-57 2024-04-16T07:45:58.138 INFO:tasks.rados.rados.0.smithi070.stdout:61: finishing write tid 4 to smithi07031500-61 2024-04-16T07:45:58.138 INFO:tasks.rados.rados.0.smithi070.stdout:57: finishing write tid 4 to smithi07031500-57 2024-04-16T07:45:58.138 INFO:tasks.rados.rados.0.smithi070.stdout:64: finishing write tid 1 to smithi07031500-64 2024-04-16T07:45:58.138 INFO:tasks.rados.rados.0.smithi070.stdout:61: finishing write tid 5 to smithi07031500-61 2024-04-16T07:45:58.138 INFO:tasks.rados.rados.0.smithi070.stdout:61: oid 61 updating version 0 to 16 2024-04-16T07:45:58.138 INFO:tasks.rados.rados.0.smithi070.stdout:61: oid 61 version 16 is already newer than 16 2024-04-16T07:45:58.138 INFO:tasks.rados.rados.0.smithi070.stdout:61: oid 61 version 16 is already newer than 16 2024-04-16T07:45:58.138 INFO:tasks.rados.rados.0.smithi070.stdout:61: oid 61 version 16 is already newer than 13 2024-04-16T07:45:58.138 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 61 v 16 (ObjNum 60 snap 0 seq_num 60) dirty exists 2024-04-16T07:45:58.139 INFO:tasks.rados.rados.0.smithi070.stdout:61: left oid 61 (ObjNum 60 snap 0 seq_num 60) 2024-04-16T07:45:58.139 INFO:tasks.rados.rados.0.smithi070.stdout:63: finishing write tid 3 to smithi07031500-63 2024-04-16T07:45:58.139 INFO:tasks.rados.rados.0.smithi070.stdout:63: finishing write tid 4 to smithi07031500-63 2024-04-16T07:45:58.139 INFO:tasks.rados.rados.0.smithi070.stdout:63: oid 63 updating version 0 to 5 2024-04-16T07:45:58.139 INFO:tasks.rados.rados.0.smithi070.stdout:63: oid 63 updating version 5 to 6 2024-04-16T07:45:58.139 INFO:tasks.rados.rados.0.smithi070.stdout:63: oid 63 version 6 is already newer than 4 2024-04-16T07:45:58.139 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 63 v 6 (ObjNum 62 snap 0 seq_num 62) dirty exists 2024-04-16T07:45:58.139 INFO:tasks.rados.rados.0.smithi070.stdout:63: left oid 63 (ObjNum 62 snap 0 seq_num 62) 2024-04-16T07:45:58.139 INFO:tasks.rados.rados.0.smithi070.stdout:66: finishing write tid 1 to smithi07031500-66 2024-04-16T07:45:58.139 INFO:tasks.rados.rados.0.smithi070.stdout:57: finishing write tid 5 to smithi07031500-57 2024-04-16T07:45:58.139 INFO:tasks.rados.rados.0.smithi070.stdout:57: oid 57 updating version 0 to 28 2024-04-16T07:45:58.139 INFO:tasks.rados.rados.0.smithi070.stdout:57: oid 57 version 28 is already newer than 25 2024-04-16T07:45:58.139 INFO:tasks.rados.rados.0.smithi070.stdout:57: oid 57 version 28 is already newer than 28 2024-04-16T07:45:58.139 INFO:tasks.rados.rados.0.smithi070.stdout:57: oid 57 version 28 is already newer than 28 2024-04-16T07:45:58.139 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 57 v 28 (ObjNum 56 snap 0 seq_num 56) dirty exists 2024-04-16T07:45:58.139 INFO:tasks.rados.rados.0.smithi070.stdout:57: left oid 57 (ObjNum 56 snap 0 seq_num 56) 2024-04-16T07:45:58.139 INFO:tasks.rados.rados.0.smithi070.stdout:57: done (15 left) 2024-04-16T07:45:58.140 INFO:tasks.rados.rados.0.smithi070.stdout:61: done (14 left) 2024-04-16T07:45:58.140 INFO:tasks.rados.rados.0.smithi070.stdout:63: done (13 left) 2024-04-16T07:45:58.140 INFO:tasks.rados.rados.0.smithi070.stdout:76: write initial oid 76 2024-04-16T07:45:58.140 INFO:tasks.rados.rados.0.smithi070.stdout:76: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2024-04-16T07:45:58.144 INFO:tasks.rados.rados.0.smithi070.stdout:76: writing smithi07031500-76 from 460525 to 1151377 tid 1 2024-04-16T07:45:58.152 INFO:tasks.rados.rados.0.smithi070.stdout:76: writing smithi07031500-76 from 1622620 to 2352972 tid 2 2024-04-16T07:45:58.158 INFO:tasks.rados.rados.0.smithi070.stdout:76: writing smithi07031500-76 from 3060489 to 3260525 tid 3 2024-04-16T07:45:58.158 INFO:tasks.rados.rados.0.smithi070.stdout:77: write initial oid 77 2024-04-16T07:45:58.158 INFO:tasks.rados.rados.0.smithi070.stdout:77: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2024-04-16T07:45:58.168 INFO:tasks.rados.rados.0.smithi070.stdout:77: writing smithi07031500-77 from 477332 to 1243433 tid 1 2024-04-16T07:45:58.175 INFO:tasks.rados.rados.0.smithi070.stdout:77: writing smithi07031500-77 from 1681102 to 2155112 tid 2 2024-04-16T07:45:58.183 INFO:tasks.rados.rados.0.smithi070.stdout:77: writing smithi07031500-77 from 2687912 to 3277332 tid 3 2024-04-16T07:45:58.184 INFO:tasks.rados.rados.0.smithi070.stdout:78: write initial oid 78 2024-04-16T07:45:58.184 INFO:tasks.rados.rados.0.smithi070.stdout:78: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2024-04-16T07:45:58.189 INFO:tasks.rados.rados.0.smithi070.stdout:78: writing smithi07031500-78 from 494139 to 1051842 tid 1 2024-04-16T07:45:58.196 INFO:tasks.rados.rados.0.smithi070.stdout:78: writing smithi07031500-78 from 1455937 to 2189958 tid 2 2024-04-16T07:45:58.203 INFO:tasks.rados.rados.0.smithi070.stdout:78: writing smithi07031500-78 from 2948041 to 3294139 tid 3 2024-04-16T07:45:58.203 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:58.203 INFO:tasks.rados.rados.0.smithi070.stdout:64: finishing write tid 2 to smithi07031500-64 2024-04-16T07:45:58.203 INFO:tasks.rados.rados.0.smithi070.stdout:68: finishing write tid 1 to smithi07031500-68 2024-04-16T07:45:58.203 INFO:tasks.rados.rados.0.smithi070.stdout:56: finishing write tid 2 to smithi07031500-56 2024-04-16T07:45:58.203 INFO:tasks.rados.rados.0.smithi070.stdout:56: finishing write tid 3 to smithi07031500-56 2024-04-16T07:45:58.204 INFO:tasks.rados.rados.0.smithi070.stdout:64: finishing write tid 3 to smithi07031500-64 2024-04-16T07:45:58.204 INFO:tasks.rados.rados.0.smithi070.stdout:64: finishing write tid 4 to smithi07031500-64 2024-04-16T07:45:58.204 INFO:tasks.rados.rados.0.smithi070.stdout:64: finishing write tid 5 to smithi07031500-64 2024-04-16T07:45:58.204 INFO:tasks.rados.rados.0.smithi070.stdout:64: oid 64 updating version 0 to 20 2024-04-16T07:45:58.204 INFO:tasks.rados.rados.0.smithi070.stdout:64: oid 64 version 20 is already newer than 17 2024-04-16T07:45:58.204 INFO:tasks.rados.rados.0.smithi070.stdout:64: oid 64 version 20 is already newer than 20 2024-04-16T07:45:58.204 INFO:tasks.rados.rados.0.smithi070.stdout:64: oid 64 version 20 is already newer than 20 2024-04-16T07:45:58.204 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 64 v 20 (ObjNum 63 snap 0 seq_num 63) dirty exists 2024-04-16T07:45:58.204 INFO:tasks.rados.rados.0.smithi070.stdout:64: left oid 64 (ObjNum 63 snap 0 seq_num 63) 2024-04-16T07:45:58.204 INFO:tasks.rados.rados.0.smithi070.stdout:73: finishing write tid 1 to smithi07031500-73 2024-04-16T07:45:58.204 INFO:tasks.rados.rados.0.smithi070.stdout:66: finishing write tid 2 to smithi07031500-66 2024-04-16T07:45:58.204 INFO:tasks.rados.rados.0.smithi070.stdout:74: finishing write tid 1 to smithi07031500-74 2024-04-16T07:45:58.204 INFO:tasks.rados.rados.0.smithi070.stdout:70: finishing write tid 1 to smithi07031500-70 2024-04-16T07:45:58.204 INFO:tasks.rados.rados.0.smithi070.stdout:66: finishing write tid 3 to smithi07031500-66 2024-04-16T07:45:58.204 INFO:tasks.rados.rados.0.smithi070.stdout:66: finishing write tid 4 to smithi07031500-66 2024-04-16T07:45:58.204 INFO:tasks.rados.rados.0.smithi070.stdout:66: oid 66 updating version 0 to 12 2024-04-16T07:45:58.204 INFO:tasks.rados.rados.0.smithi070.stdout:66: oid 66 version 12 is already newer than 10 2024-04-16T07:45:58.205 INFO:tasks.rados.rados.0.smithi070.stdout:66: oid 66 version 12 is already newer than 12 2024-04-16T07:45:58.205 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 66 v 12 (ObjNum 65 snap 0 seq_num 65) dirty exists 2024-04-16T07:45:58.205 INFO:tasks.rados.rados.0.smithi070.stdout:66: left oid 66 (ObjNum 65 snap 0 seq_num 65) 2024-04-16T07:45:58.205 INFO:tasks.rados.rados.0.smithi070.stdout:68: finishing write tid 2 to smithi07031500-68 2024-04-16T07:45:58.205 INFO:tasks.rados.rados.0.smithi070.stdout:64: done (15 left) 2024-04-16T07:45:58.205 INFO:tasks.rados.rados.0.smithi070.stdout:66: done (14 left) 2024-04-16T07:45:58.205 INFO:tasks.rados.rados.0.smithi070.stdout:79: write initial oid 79 2024-04-16T07:45:58.205 INFO:tasks.rados.rados.0.smithi070.stdout:79: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2024-04-16T07:45:58.212 INFO:tasks.rados.rados.0.smithi070.stdout:79: writing smithi07031500-79 from 510946 to 1143898 tid 1 2024-04-16T07:45:58.219 INFO:tasks.rados.rados.0.smithi070.stdout:79: writing smithi07031500-79 from 1798066 to 2275745 tid 2 2024-04-16T07:45:58.226 INFO:tasks.rados.rados.0.smithi070.stdout:79: writing smithi07031500-79 from 2742758 to 3199975 tid 3 2024-04-16T07:45:58.226 INFO:tasks.rados.rados.0.smithi070.stdout:80: write initial oid 80 2024-04-16T07:45:58.226 INFO:tasks.rados.rados.0.smithi070.stdout:80: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2024-04-16T07:45:58.233 INFO:tasks.rados.rados.0.smithi070.stdout:80: writing smithi07031500-80 from 527753 to 1235954 tid 1 2024-04-16T07:45:58.241 INFO:tasks.rados.rados.0.smithi070.stdout:80: writing smithi07031500-80 from 1856548 to 2594238 tid 2 2024-04-16T07:45:58.246 INFO:tasks.rados.rados.0.smithi070.stdout:80: writing smithi07031500-80 from 3286534 to 3327753 tid 3 2024-04-16T07:45:58.247 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:58.247 INFO:tasks.rados.rados.0.smithi070.stdout:56: finishing write tid 4 to smithi07031500-56 2024-04-16T07:45:58.247 INFO:tasks.rados.rados.0.smithi070.stdout:71: finishing write tid 1 to smithi07031500-71 2024-04-16T07:45:58.247 INFO:tasks.rados.rados.0.smithi070.stdout:56: finishing write tid 5 to smithi07031500-56 2024-04-16T07:45:58.247 INFO:tasks.rados.rados.0.smithi070.stdout:56: oid 56 updating version 0 to 16 2024-04-16T07:45:58.247 INFO:tasks.rados.rados.0.smithi070.stdout:56: oid 56 updating version 16 to 19 2024-04-16T07:45:58.247 INFO:tasks.rados.rados.0.smithi070.stdout:56: oid 56 version 19 is already newer than 19 2024-04-16T07:45:58.247 INFO:tasks.rados.rados.0.smithi070.stdout:56: oid 56 version 19 is already newer than 19 2024-04-16T07:45:58.247 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 56 v 19 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-04-16T07:45:58.247 INFO:tasks.rados.rados.0.smithi070.stdout:56: left oid 56 (ObjNum 55 snap 0 seq_num 55) 2024-04-16T07:45:58.247 INFO:tasks.rados.rados.0.smithi070.stdout:72: finishing write tid 1 to smithi07031500-72 2024-04-16T07:45:58.247 INFO:tasks.rados.rados.0.smithi070.stdout:56: done (15 left) 2024-04-16T07:45:58.247 INFO:tasks.rados.rados.0.smithi070.stdout:81: write initial oid 81 2024-04-16T07:45:58.247 INFO:tasks.rados.rados.0.smithi070.stdout:81: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-04-16T07:45:58.256 INFO:tasks.rados.rados.0.smithi070.stdout:81: writing smithi07031500-81 from 544560 to 1328010 tid 1 2024-04-16T07:45:58.264 INFO:tasks.rados.rados.0.smithi070.stdout:81: writing smithi07031500-81 from 1915030 to 2396378 tid 2 2024-04-16T07:45:58.269 INFO:tasks.rados.rados.0.smithi070.stdout:81: writing smithi07031500-81 from 2797604 to 3277365 tid 3 2024-04-16T07:45:58.269 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:58.269 INFO:tasks.rados.rados.0.smithi070.stdout:70: finishing write tid 2 to smithi07031500-70 2024-04-16T07:45:58.269 INFO:tasks.rados.rados.0.smithi070.stdout:60: finishing write tid 1 to smithi07031500-60 2024-04-16T07:45:58.270 INFO:tasks.rados.rados.0.smithi070.stdout:77: finishing write tid 1 to smithi07031500-77 2024-04-16T07:45:58.270 INFO:tasks.rados.rados.0.smithi070.stdout:67: finishing write tid 1 to smithi07031500-67 2024-04-16T07:45:58.270 INFO:tasks.rados.rados.0.smithi070.stdout:70: finishing write tid 3 to smithi07031500-70 2024-04-16T07:45:58.273 INFO:tasks.rados.rados.0.smithi070.stdout:71: finishing write tid 2 to smithi07031500-71 2024-04-16T07:45:58.279 INFO:tasks.rados.rados.0.smithi070.stdout:73: finishing write tid 2 to smithi07031500-73 2024-04-16T07:45:58.279 INFO:tasks.rados.rados.0.smithi070.stdout:70: finishing write tid 4 to smithi07031500-70 2024-04-16T07:45:58.286 INFO:tasks.rados.rados.0.smithi070.stdout:70: finishing write tid 5 to smithi07031500-70 2024-04-16T07:45:58.286 INFO:tasks.rados.rados.0.smithi070.stdout:70: oid 70 updating version 0 to 32 2024-04-16T07:45:58.286 INFO:tasks.rados.rados.0.smithi070.stdout:70: oid 70 version 32 is already newer than 32 2024-04-16T07:45:58.286 INFO:tasks.rados.rados.0.smithi070.stdout:70: oid 70 version 32 is already newer than 32 2024-04-16T07:45:58.286 INFO:tasks.rados.rados.0.smithi070.stdout:70: oid 70 version 32 is already newer than 29 2024-04-16T07:45:58.286 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 70 v 32 (ObjNum 69 snap 0 seq_num 69) dirty exists 2024-04-16T07:45:58.286 INFO:tasks.rados.rados.0.smithi070.stdout:70: left oid 70 (ObjNum 69 snap 0 seq_num 69) 2024-04-16T07:45:58.286 INFO:tasks.rados.rados.0.smithi070.stdout:70: done (15 left) 2024-04-16T07:45:58.286 INFO:tasks.rados.rados.0.smithi070.stdout:82: write initial oid 82 2024-04-16T07:45:58.286 INFO:tasks.rados.rados.0.smithi070.stdout:82: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-04-16T07:45:58.292 INFO:tasks.rados.rados.0.smithi070.stdout:82: writing smithi07031500-82 from 561367 to 1020066 tid 1 2024-04-16T07:45:58.300 INFO:tasks.rados.rados.0.smithi070.stdout:82: writing smithi07031500-82 from 1573512 to 2314871 tid 2 2024-04-16T07:45:58.308 INFO:tasks.rados.rados.0.smithi070.stdout:82: writing smithi07031500-82 from 2941380 to 3361367 tid 3 2024-04-16T07:45:58.308 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:58.308 INFO:tasks.rados.rados.0.smithi070.stdout:68: finishing write tid 3 to smithi07031500-68 2024-04-16T07:45:58.308 INFO:tasks.rados.rados.0.smithi070.stdout:74: finishing write tid 2 to smithi07031500-74 2024-04-16T07:45:58.308 INFO:tasks.rados.rados.0.smithi070.stdout:68: finishing write tid 4 to smithi07031500-68 2024-04-16T07:45:58.308 INFO:tasks.rados.rados.0.smithi070.stdout:68: oid 68 updating version 0 to 12 2024-04-16T07:45:58.308 INFO:tasks.rados.rados.0.smithi070.stdout:68: oid 68 version 12 is already newer than 11 2024-04-16T07:45:58.308 INFO:tasks.rados.rados.0.smithi070.stdout:68: oid 68 version 12 is already newer than 10 2024-04-16T07:45:58.308 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 68 v 12 (ObjNum 67 snap 0 seq_num 67) dirty exists 2024-04-16T07:45:58.308 INFO:tasks.rados.rados.0.smithi070.stdout:68: left oid 68 (ObjNum 67 snap 0 seq_num 67) 2024-04-16T07:45:58.308 INFO:tasks.rados.rados.0.smithi070.stdout:68: done (15 left) 2024-04-16T07:45:58.309 INFO:tasks.rados.rados.0.smithi070.stdout:83: write initial oid 83 2024-04-16T07:45:58.309 INFO:tasks.rados.rados.0.smithi070.stdout:83: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2024-04-16T07:45:58.314 INFO:tasks.rados.rados.0.smithi070.stdout:83: writing smithi07031500-83 from 578174 to 1112122 tid 1 2024-04-16T07:45:58.321 INFO:tasks.rados.rados.0.smithi070.stdout:83: writing smithi07031500-83 from 1515641 to 2000658 tid 2 2024-04-16T07:45:58.331 INFO:tasks.rados.rados.0.smithi070.stdout:83: writing smithi07031500-83 from 2736097 to 3238402 tid 3 2024-04-16T07:45:58.331 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:58.331 INFO:tasks.rados.rados.0.smithi070.stdout:72: finishing write tid 2 to smithi07031500-72 2024-04-16T07:45:58.331 INFO:tasks.rados.rados.0.smithi070.stdout:71: finishing write tid 3 to smithi07031500-71 2024-04-16T07:45:58.331 INFO:tasks.rados.rados.0.smithi070.stdout:60: finishing write tid 2 to smithi07031500-60 2024-04-16T07:45:58.331 INFO:tasks.rados.rados.0.smithi070.stdout:71: finishing write tid 4 to smithi07031500-71 2024-04-16T07:45:58.331 INFO:tasks.rados.rados.0.smithi070.stdout:71: oid 71 updating version 0 to 19 2024-04-16T07:45:58.331 INFO:tasks.rados.rados.0.smithi070.stdout:71: oid 71 version 19 is already newer than 19 2024-04-16T07:45:58.331 INFO:tasks.rados.rados.0.smithi070.stdout:71: oid 71 version 19 is already newer than 17 2024-04-16T07:45:58.331 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 71 v 19 (ObjNum 70 snap 0 seq_num 70) dirty exists 2024-04-16T07:45:58.332 INFO:tasks.rados.rados.0.smithi070.stdout:71: left oid 71 (ObjNum 70 snap 0 seq_num 70) 2024-04-16T07:45:58.332 INFO:tasks.rados.rados.0.smithi070.stdout:72: finishing write tid 3 to smithi07031500-72 2024-04-16T07:45:58.332 INFO:tasks.rados.rados.0.smithi070.stdout:72: finishing write tid 4 to smithi07031500-72 2024-04-16T07:45:58.332 INFO:tasks.rados.rados.0.smithi070.stdout:72: oid 72 updating version 0 to 21 2024-04-16T07:45:58.332 INFO:tasks.rados.rados.0.smithi070.stdout:72: oid 72 updating version 21 to 23 2024-04-16T07:45:58.332 INFO:tasks.rados.rados.0.smithi070.stdout:72: oid 72 version 23 is already newer than 23 2024-04-16T07:45:58.332 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 72 v 23 (ObjNum 71 snap 0 seq_num 71) dirty exists 2024-04-16T07:45:58.332 INFO:tasks.rados.rados.0.smithi070.stdout:72: left oid 72 (ObjNum 71 snap 0 seq_num 71) 2024-04-16T07:45:58.332 INFO:tasks.rados.rados.0.smithi070.stdout:71: done (15 left) 2024-04-16T07:45:58.332 INFO:tasks.rados.rados.0.smithi070.stdout:72: done (14 left) 2024-04-16T07:45:58.332 INFO:tasks.rados.rados.0.smithi070.stdout:84: write initial oid 84 2024-04-16T07:45:58.332 INFO:tasks.rados.rados.0.smithi070.stdout:84: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-04-16T07:45:58.339 INFO:tasks.rados.rados.0.smithi070.stdout:84: writing smithi07031500-84 from 594981 to 1204178 tid 1 2024-04-16T07:45:58.349 INFO:tasks.rados.rados.0.smithi070.stdout:84: writing smithi07031500-84 from 1974123 to 2719151 tid 2 2024-04-16T07:45:58.354 INFO:tasks.rados.rados.0.smithi070.stdout:84: writing smithi07031500-84 from 3279873 to 3394981 tid 3 2024-04-16T07:45:58.354 INFO:tasks.rados.rados.0.smithi070.stdout:85: write initial oid 85 2024-04-16T07:45:58.354 INFO:tasks.rados.rados.0.smithi070.stdout:85: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-04-16T07:45:58.360 INFO:tasks.rados.rados.0.smithi070.stdout:85: writing smithi07031500-85 from 611788 to 1012587 tid 1 2024-04-16T07:45:58.367 INFO:tasks.rados.rados.0.smithi070.stdout:85: writing smithi07031500-85 from 1748958 to 2237644 tid 2 2024-04-16T07:45:58.375 INFO:tasks.rados.rados.0.smithi070.stdout:85: writing smithi07031500-85 from 2907296 to 3411788 tid 3 2024-04-16T07:45:58.375 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:58.375 INFO:tasks.rados.rados.0.smithi070.stdout:67: finishing write tid 2 to smithi07031500-67 2024-04-16T07:45:58.375 INFO:tasks.rados.rados.0.smithi070.stdout:60: finishing write tid 3 to smithi07031500-60 2024-04-16T07:45:58.375 INFO:tasks.rados.rados.0.smithi070.stdout:60: finishing write tid 4 to smithi07031500-60 2024-04-16T07:45:58.375 INFO:tasks.rados.rados.0.smithi070.stdout:60: finishing write tid 5 to smithi07031500-60 2024-04-16T07:45:58.375 INFO:tasks.rados.rados.0.smithi070.stdout:60: oid 60 updating version 0 to 23 2024-04-16T07:45:58.375 INFO:tasks.rados.rados.0.smithi070.stdout:60: oid 60 version 23 is already newer than 23 2024-04-16T07:45:58.375 INFO:tasks.rados.rados.0.smithi070.stdout:60: oid 60 version 23 is already newer than 23 2024-04-16T07:45:58.375 INFO:tasks.rados.rados.0.smithi070.stdout:60: oid 60 version 23 is already newer than 20 2024-04-16T07:45:58.375 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 60 v 23 (ObjNum 59 snap 0 seq_num 59) dirty exists 2024-04-16T07:45:58.375 INFO:tasks.rados.rados.0.smithi070.stdout:60: left oid 60 (ObjNum 59 snap 0 seq_num 59) 2024-04-16T07:45:58.375 INFO:tasks.rados.rados.0.smithi070.stdout:67: finishing write tid 3 to smithi07031500-67 2024-04-16T07:45:58.376 INFO:tasks.rados.rados.0.smithi070.stdout:77: finishing write tid 2 to smithi07031500-77 2024-04-16T07:45:58.376 INFO:tasks.rados.rados.0.smithi070.stdout:67: finishing write tid 4 to smithi07031500-67 2024-04-16T07:45:58.376 INFO:tasks.rados.rados.0.smithi070.stdout:67: finishing write tid 5 to smithi07031500-67 2024-04-16T07:45:58.376 INFO:tasks.rados.rados.0.smithi070.stdout:67: oid 67 updating version 0 to 16 2024-04-16T07:45:58.376 INFO:tasks.rados.rados.0.smithi070.stdout:67: oid 67 version 16 is already newer than 13 2024-04-16T07:45:58.376 INFO:tasks.rados.rados.0.smithi070.stdout:67: oid 67 version 16 is already newer than 16 2024-04-16T07:45:58.376 INFO:tasks.rados.rados.0.smithi070.stdout:67: oid 67 version 16 is already newer than 16 2024-04-16T07:45:58.376 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 67 v 16 (ObjNum 66 snap 0 seq_num 66) dirty exists 2024-04-16T07:45:58.376 INFO:tasks.rados.rados.0.smithi070.stdout:67: left oid 67 (ObjNum 66 snap 0 seq_num 66) 2024-04-16T07:45:58.376 INFO:tasks.rados.rados.0.smithi070.stdout:83: finishing write tid 1 to smithi07031500-83 2024-04-16T07:45:58.376 INFO:tasks.rados.rados.0.smithi070.stdout:77: finishing write tid 3 to smithi07031500-77 2024-04-16T07:45:58.376 INFO:tasks.rados.rados.0.smithi070.stdout:60: done (15 left) 2024-04-16T07:45:58.376 INFO:tasks.rados.rados.0.smithi070.stdout:67: done (14 left) 2024-04-16T07:45:58.376 INFO:tasks.rados.rados.0.smithi070.stdout:86: write initial oid 86 2024-04-16T07:45:58.376 INFO:tasks.rados.rados.0.smithi070.stdout:86: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-04-16T07:45:58.380 INFO:tasks.rados.rados.0.smithi070.stdout:86: writing smithi07031500-86 from 628595 to 1104643 tid 1 2024-04-16T07:45:58.391 INFO:tasks.rados.rados.0.smithi070.stdout:86: writing smithi07031500-86 from 1807440 to 2439784 tid 2 2024-04-16T07:45:58.398 INFO:tasks.rados.rados.0.smithi070.stdout:86: writing smithi07031500-86 from 2934719 to 3387193 tid 3 2024-04-16T07:45:58.398 INFO:tasks.rados.rados.0.smithi070.stdout:87: write initial oid 87 2024-04-16T07:45:58.399 INFO:tasks.rados.rados.0.smithi070.stdout:87: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2024-04-16T07:45:58.404 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-16T07:45:58.404 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:45:58.404 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2, 3] 2024-04-16T07:45:58.404 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-16T07:45:58.405 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:45:58.407 INFO:tasks.rados.rados.0.smithi070.stdout:87: writing smithi07031500-87 from 645402 to 1196699 tid 1 2024-04-16T07:45:58.414 INFO:tasks.rados.rados.0.smithi070.stdout:87: writing smithi07031500-87 from 1749569 to 2241924 tid 2 2024-04-16T07:45:58.421 INFO:tasks.rados.rados.0.smithi070.stdout:87: writing smithi07031500-87 from 2845789 to 3445402 tid 3 2024-04-16T07:45:58.421 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:58.421 INFO:tasks.rados.rados.0.smithi070.stdout:69: finishing write tid 1 to smithi07031500-69 2024-04-16T07:45:58.422 INFO:tasks.rados.rados.0.smithi070.stdout:78: finishing write tid 1 to smithi07031500-78 2024-04-16T07:45:58.422 INFO:tasks.rados.rados.0.smithi070.stdout:73: finishing write tid 3 to smithi07031500-73 2024-04-16T07:45:58.422 INFO:tasks.rados.rados.0.smithi070.stdout:82: finishing write tid 1 to smithi07031500-82 2024-04-16T07:45:58.422 INFO:tasks.rados.rados.0.smithi070.stdout:81: finishing write tid 1 to smithi07031500-81 2024-04-16T07:45:58.422 INFO:tasks.rados.rados.0.smithi070.stdout:83: finishing write tid 2 to smithi07031500-83 2024-04-16T07:45:58.422 INFO:tasks.rados.rados.0.smithi070.stdout:74: finishing write tid 3 to smithi07031500-74 2024-04-16T07:45:58.422 INFO:tasks.rados.rados.0.smithi070.stdout:84: finishing write tid 1 to smithi07031500-84 2024-04-16T07:45:58.432 INFO:tasks.rados.rados.0.smithi070.stdout:76: finishing write tid 1 to smithi07031500-76 2024-04-16T07:45:58.440 INFO:tasks.rados.rados.0.smithi070.stdout:77: finishing write tid 4 to smithi07031500-77 2024-04-16T07:45:58.443 INFO:tasks.rados.rados.0.smithi070.stdout:77: finishing write tid 5 to smithi07031500-77 2024-04-16T07:45:58.444 INFO:tasks.rados.rados.0.smithi070.stdout:77: oid 77 updating version 0 to 13 2024-04-16T07:45:58.444 INFO:tasks.rados.rados.0.smithi070.stdout:77: oid 77 version 13 is already newer than 11 2024-04-16T07:45:58.444 INFO:tasks.rados.rados.0.smithi070.stdout:77: oid 77 version 13 is already newer than 13 2024-04-16T07:45:58.444 INFO:tasks.rados.rados.0.smithi070.stdout:77: oid 77 updating version 13 to 14 2024-04-16T07:45:58.444 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 77 v 14 (ObjNum 76 snap 0 seq_num 76) dirty exists 2024-04-16T07:45:58.444 INFO:tasks.rados.rados.0.smithi070.stdout:77: left oid 77 (ObjNum 76 snap 0 seq_num 76) 2024-04-16T07:45:58.444 INFO:tasks.rados.rados.0.smithi070.stdout:77: done (15 left) 2024-04-16T07:45:58.444 INFO:tasks.rados.rados.0.smithi070.stdout:88: write initial oid 88 2024-04-16T07:45:58.444 INFO:tasks.rados.rados.0.smithi070.stdout:88: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2024-04-16T07:45:58.450 INFO:tasks.rados.rados.0.smithi070.stdout:88: writing smithi07031500-88 from 662209 to 1288755 tid 1 2024-04-16T07:45:58.457 INFO:tasks.rados.rados.0.smithi070.stdout:88: writing smithi07031500-88 from 1808051 to 2444064 tid 2 2024-04-16T07:45:58.463 INFO:tasks.rados.rados.0.smithi070.stdout:88: writing smithi07031500-88 from 2873212 to 3462209 tid 3 2024-04-16T07:45:58.463 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:58.463 INFO:tasks.rados.rados.0.smithi070.stdout:69: finishing write tid 2 to smithi07031500-69 2024-04-16T07:45:58.463 INFO:tasks.rados.rados.0.smithi070.stdout:82: finishing write tid 2 to smithi07031500-82 2024-04-16T07:45:58.478 INFO:tasks.ceph.osd.0:Stopped 2024-04-16T07:45:58.480 INFO:tasks.rados.rados.0.smithi070.stdout:84: finishing write tid 2 to smithi07031500-84 2024-04-16T07:45:58.502 INFO:tasks.rados.rados.0.smithi070.stdout:79: finishing write tid 1 to smithi07031500-79 2024-04-16T07:45:58.523 INFO:tasks.rados.rados.0.smithi070.stdout:82: finishing write tid 3 to smithi07031500-82 2024-04-16T07:45:58.533 INFO:tasks.rados.rados.0.smithi070.stdout:84: finishing write tid 3 to smithi07031500-84 2024-04-16T07:45:58.536 INFO:tasks.rados.rados.0.smithi070.stdout:79: finishing write tid 2 to smithi07031500-79 2024-04-16T07:45:58.539 INFO:tasks.rados.rados.0.smithi070.stdout:84: finishing write tid 4 to smithi07031500-84 2024-04-16T07:45:58.541 INFO:tasks.rados.rados.0.smithi070.stdout:82: finishing write tid 4 to smithi07031500-82 2024-04-16T07:45:58.545 INFO:tasks.rados.rados.0.smithi070.stdout:84: finishing write tid 5 to smithi07031500-84 2024-04-16T07:45:58.545 INFO:tasks.rados.rados.0.smithi070.stdout:84: oid 84 updating version 0 to 19 2024-04-16T07:45:58.545 INFO:tasks.rados.rados.0.smithi070.stdout:84: oid 84 version 19 is already newer than 19 2024-04-16T07:45:58.545 INFO:tasks.rados.rados.0.smithi070.stdout:84: oid 84 version 19 is already newer than 17 2024-04-16T07:45:58.545 INFO:tasks.rados.rados.0.smithi070.stdout:84: oid 84 version 19 is already newer than 16 2024-04-16T07:45:58.545 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 84 v 19 (ObjNum 83 snap 0 seq_num 83) dirty exists 2024-04-16T07:45:58.545 INFO:tasks.rados.rados.0.smithi070.stdout:84: left oid 84 (ObjNum 83 snap 0 seq_num 83) 2024-04-16T07:45:58.545 INFO:tasks.rados.rados.0.smithi070.stdout:84: done (15 left) 2024-04-16T07:45:58.545 INFO:tasks.rados.rados.0.smithi070.stdout:89: write initial oid 89 2024-04-16T07:45:58.545 INFO:tasks.rados.rados.0.smithi070.stdout:89: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-04-16T07:45:58.552 INFO:tasks.rados.rados.0.smithi070.stdout:89: writing smithi07031500-89 from 679016 to 1380811 tid 1 2024-04-16T07:45:58.558 INFO:tasks.rados.rados.0.smithi070.stdout:89: writing smithi07031500-89 from 1866533 to 2362557 tid 2 2024-04-16T07:45:58.564 INFO:tasks.rados.rados.0.smithi070.stdout:89: writing smithi07031500-89 from 2900635 to 3303278 tid 3 2024-04-16T07:45:58.564 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:58.564 INFO:tasks.rados.rados.0.smithi070.stdout:82: finishing write tid 5 to smithi07031500-82 2024-04-16T07:45:58.564 INFO:tasks.rados.rados.0.smithi070.stdout:82: oid 82 updating version 0 to 23 2024-04-16T07:45:58.564 INFO:tasks.rados.rados.0.smithi070.stdout:82: oid 82 version 23 is already newer than 21 2024-04-16T07:45:58.564 INFO:tasks.rados.rados.0.smithi070.stdout:82: oid 82 version 23 is already newer than 20 2024-04-16T07:45:58.564 INFO:tasks.rados.rados.0.smithi070.stdout:82: oid 82 version 23 is already newer than 22 2024-04-16T07:45:58.564 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 82 v 23 (ObjNum 81 snap 0 seq_num 81) dirty exists 2024-04-16T07:45:58.564 INFO:tasks.rados.rados.0.smithi070.stdout:82: left oid 82 (ObjNum 81 snap 0 seq_num 81) 2024-04-16T07:45:58.564 INFO:tasks.rados.rados.0.smithi070.stdout:82: done (15 left) 2024-04-16T07:45:58.564 INFO:tasks.rados.rados.0.smithi070.stdout:90: write initial oid 90 2024-04-16T07:45:58.564 INFO:tasks.rados.rados.0.smithi070.stdout:90: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2024-04-16T07:45:58.571 INFO:tasks.rados.rados.0.smithi070.stdout:90: writing smithi07031500-90 from 695823 to 1472867 tid 1 2024-04-16T07:45:58.577 INFO:tasks.rados.rados.0.smithi070.stdout:90: writing smithi07031500-90 from 1925015 to 2564697 tid 2 2024-04-16T07:45:58.583 INFO:tasks.rados.rados.0.smithi070.stdout:90: writing smithi07031500-90 from 3328058 to 3495823 tid 3 2024-04-16T07:45:58.583 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:58.583 INFO:tasks.rados.rados.0.smithi070.stdout:79: finishing write tid 3 to smithi07031500-79 2024-04-16T07:45:58.583 INFO:tasks.rados.rados.0.smithi070.stdout:79: finishing write tid 4 to smithi07031500-79 2024-04-16T07:45:58.583 INFO:tasks.rados.rados.0.smithi070.stdout:79: finishing write tid 5 to smithi07031500-79 2024-04-16T07:45:58.583 INFO:tasks.rados.rados.0.smithi070.stdout:79: oid 79 updating version 0 to 18 2024-04-16T07:45:58.583 INFO:tasks.rados.rados.0.smithi070.stdout:79: oid 79 version 18 is already newer than 15 2024-04-16T07:45:58.583 INFO:tasks.rados.rados.0.smithi070.stdout:79: oid 79 version 18 is already newer than 17 2024-04-16T07:45:58.583 INFO:tasks.rados.rados.0.smithi070.stdout:79: oid 79 version 18 is already newer than 16 2024-04-16T07:45:58.583 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 79 v 18 (ObjNum 78 snap 0 seq_num 78) dirty exists 2024-04-16T07:45:58.583 INFO:tasks.rados.rados.0.smithi070.stdout:79: left oid 79 (ObjNum 78 snap 0 seq_num 78) 2024-04-16T07:45:58.583 INFO:tasks.rados.rados.0.smithi070.stdout:79: done (15 left) 2024-04-16T07:45:58.584 INFO:tasks.rados.rados.0.smithi070.stdout:91: write initial oid 91 2024-04-16T07:45:58.584 INFO:tasks.rados.rados.0.smithi070.stdout:91: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2024-04-16T07:45:58.588 INFO:tasks.rados.rados.0.smithi070.stdout:91: writing smithi07031500-91 from 712630 to 1164923 tid 1 2024-04-16T07:45:58.593 INFO:tasks.rados.rados.0.smithi070.stdout:91: writing smithi07031500-91 from 1583497 to 2083190 tid 2 2024-04-16T07:45:58.598 INFO:tasks.rados.rados.0.smithi070.stdout:91: writing smithi07031500-91 from 2555481 to 2980668 tid 3 2024-04-16T07:45:58.598 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:45:58.609 INFO:tasks.rados.rados.0.smithi070.stdout:91: finishing write tid 1 to smithi07031500-91 2024-04-16T07:45:58.623 INFO:tasks.rados.rados.0.smithi070.stdout:91: finishing write tid 2 to smithi07031500-91 2024-04-16T07:45:58.642 INFO:tasks.rados.rados.0.smithi070.stdout:91: finishing write tid 3 to smithi07031500-91 2024-04-16T07:45:58.650 INFO:tasks.rados.rados.0.smithi070.stdout:91: finishing write tid 4 to smithi07031500-91 2024-04-16T07:45:58.652 INFO:tasks.rados.rados.0.smithi070.stdout:91: finishing write tid 5 to smithi07031500-91 2024-04-16T07:45:58.652 INFO:tasks.rados.rados.0.smithi070.stdout:91: oid 91 updating version 0 to 26 2024-04-16T07:45:58.652 INFO:tasks.rados.rados.0.smithi070.stdout:91: oid 91 version 26 is already newer than 25 2024-04-16T07:45:58.652 INFO:tasks.rados.rados.0.smithi070.stdout:91: oid 91 version 26 is already newer than 24 2024-04-16T07:45:58.652 INFO:tasks.rados.rados.0.smithi070.stdout:91: oid 91 updating version 26 to 27 2024-04-16T07:45:58.652 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 91 v 27 (ObjNum 90 snap 0 seq_num 90) dirty exists 2024-04-16T07:45:58.652 INFO:tasks.rados.rados.0.smithi070.stdout:91: left oid 91 (ObjNum 90 snap 0 seq_num 90) 2024-04-16T07:45:58.653 INFO:tasks.rados.rados.0.smithi070.stdout:91: done (15 left) 2024-04-16T07:45:58.653 INFO:tasks.rados.rados.0.smithi070.stdout:92: write initial oid 92 2024-04-16T07:45:58.653 INFO:tasks.rados.rados.0.smithi070.stdout:92: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2024-04-16T07:45:58.658 INFO:tasks.rados.rados.0.smithi070.stdout:92: writing smithi07031500-92 from 729437 to 1256979 tid 1 2024-04-16T07:45:58.665 INFO:tasks.rados.rados.0.smithi070.stdout:92: writing smithi07031500-92 from 1925626 to 2568977 tid 2 2024-04-16T07:45:58.671 INFO:tasks.rados.rados.0.smithi070.stdout:92: writing smithi07031500-92 from 3266551 to 3529437 tid 3 2024-04-16T07:45:58.671 INFO:tasks.rados.rados.0.smithi070.stdout: waiting on 16 2024-04-16T07:46:00.353 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:46:00.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:00.473+0000 7f1ddacad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:00.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:00.478+0000 7f1ddacad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:03.479 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3] 2024-04-16T07:46:03.479 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:46:03.479 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-04-16T07:46:03.479 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-16T07:46:03.479 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-16T07:46:03.483 INFO:tasks.ceph.osd.0:Started 2024-04-16T07:46:03.483 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-16T07:46:03.596 INFO:teuthology.orchestra.run.smithi070.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:46:03.600 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:03.600 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-16T07:46:07.683 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:46:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:46:07.958 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:46:08.601 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-16T07:46:08.784 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-16T07:46:08.784 INFO:teuthology.orchestra.run.smithi070.stdout: "ops_in_flight": [ 2024-04-16T07:46:08.784 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.784 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "peering_event", 2024-04-16T07:46:08.784 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 36, 2024-04-16T07:46:08.784 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.784 INFO:teuthology.orchestra.run.smithi070.stdout: "PeeringEvent": { 2024-04-16T07:46:08.784 INFO:teuthology.orchestra.run.smithi070.stdout: "from": "0", 2024-04-16T07:46:08.784 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "1.0", 2024-04-16T07:46:08.785 INFO:teuthology.orchestra.run.smithi070.stdout: "sent": 19, 2024-04-16T07:46:08.785 INFO:teuthology.orchestra.run.smithi070.stdout: "requested": 19, 2024-04-16T07:46:08.785 INFO:teuthology.orchestra.run.smithi070.stdout: "evt": "epoch_sent: 19 epoch_requested: 19 IntervalFlush", 2024-04-16T07:46:08.785 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-04-16T07:46:08.785 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.785 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:08.785 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.785 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.785 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.785 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T07:46:08.785 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.735559+0000" 2024-04-16T07:46:08.785 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.785 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.785 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T07:46:08.786 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.786 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.786 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.786 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T07:46:08.786 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.737298+0000" 2024-04-16T07:46:08.786 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.786 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.786 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:08.786 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.786 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.786 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-16T07:46:08.786 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.786 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.786 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.787 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.787 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "peering_event", 2024-04-16T07:46:08.787 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 37, 2024-04-16T07:46:08.787 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.787 INFO:teuthology.orchestra.run.smithi070.stdout: "PeeringEvent": { 2024-04-16T07:46:08.787 INFO:teuthology.orchestra.run.smithi070.stdout: "from": "0", 2024-04-16T07:46:08.787 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "3.1", 2024-04-16T07:46:08.787 INFO:teuthology.orchestra.run.smithi070.stdout: "sent": 19, 2024-04-16T07:46:08.787 INFO:teuthology.orchestra.run.smithi070.stdout: "requested": 19, 2024-04-16T07:46:08.787 INFO:teuthology.orchestra.run.smithi070.stdout: "evt": "epoch_sent: 19 epoch_requested: 19 IntervalFlush", 2024-04-16T07:46:08.787 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-04-16T07:46:08.787 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.787 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:08.787 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.788 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.788 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.788 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T07:46:08.788 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.735097+0000" 2024-04-16T07:46:08.788 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.788 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.788 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T07:46:08.788 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.788 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.788 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.788 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T07:46:08.788 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.737265+0000" 2024-04-16T07:46:08.788 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.788 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.789 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:08.789 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.789 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.789 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-16T07:46:08.789 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.789 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.789 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.789 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.789 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "peering_event", 2024-04-16T07:46:08.789 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 38, 2024-04-16T07:46:08.789 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.789 INFO:teuthology.orchestra.run.smithi070.stdout: "PeeringEvent": { 2024-04-16T07:46:08.789 INFO:teuthology.orchestra.run.smithi070.stdout: "from": "0", 2024-04-16T07:46:08.789 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "2.6", 2024-04-16T07:46:08.789 INFO:teuthology.orchestra.run.smithi070.stdout: "sent": 19, 2024-04-16T07:46:08.790 INFO:teuthology.orchestra.run.smithi070.stdout: "requested": 19, 2024-04-16T07:46:08.790 INFO:teuthology.orchestra.run.smithi070.stdout: "evt": "epoch_sent: 19 epoch_requested: 19 IntervalFlush", 2024-04-16T07:46:08.790 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-04-16T07:46:08.790 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.790 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:08.790 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.790 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.790 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.790 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T07:46:08.790 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.735356+0000" 2024-04-16T07:46:08.790 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.790 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.790 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T07:46:08.790 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.791 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.791 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.791 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T07:46:08.791 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.737086+0000" 2024-04-16T07:46:08.791 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.791 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.791 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:08.791 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.791 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.791 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-16T07:46:08.791 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.791 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.791 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.791 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.791 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "peering_event", 2024-04-16T07:46:08.791 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 39, 2024-04-16T07:46:08.792 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.792 INFO:teuthology.orchestra.run.smithi070.stdout: "PeeringEvent": { 2024-04-16T07:46:08.792 INFO:teuthology.orchestra.run.smithi070.stdout: "from": "0", 2024-04-16T07:46:08.792 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "2.3", 2024-04-16T07:46:08.792 INFO:teuthology.orchestra.run.smithi070.stdout: "sent": 19, 2024-04-16T07:46:08.792 INFO:teuthology.orchestra.run.smithi070.stdout: "requested": 19, 2024-04-16T07:46:08.792 INFO:teuthology.orchestra.run.smithi070.stdout: "evt": "epoch_sent: 19 epoch_requested: 19 IntervalFlush", 2024-04-16T07:46:08.792 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-04-16T07:46:08.792 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.792 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:08.792 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.792 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.792 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.792 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T07:46:08.793 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.744322+0000" 2024-04-16T07:46:08.793 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.793 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.793 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T07:46:08.793 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.793 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.793 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.793 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T07:46:08.793 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.748640+0000" 2024-04-16T07:46:08.793 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.793 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.793 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:08.793 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.793 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.793 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-16T07:46:08.794 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.794 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.794 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.794 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.794 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "peering_event", 2024-04-16T07:46:08.794 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 40, 2024-04-16T07:46:08.794 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.794 INFO:teuthology.orchestra.run.smithi070.stdout: "PeeringEvent": { 2024-04-16T07:46:08.794 INFO:teuthology.orchestra.run.smithi070.stdout: "from": "0", 2024-04-16T07:46:08.794 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "3.3", 2024-04-16T07:46:08.794 INFO:teuthology.orchestra.run.smithi070.stdout: "sent": 19, 2024-04-16T07:46:08.794 INFO:teuthology.orchestra.run.smithi070.stdout: "requested": 19, 2024-04-16T07:46:08.794 INFO:teuthology.orchestra.run.smithi070.stdout: "evt": "epoch_sent: 19 epoch_requested: 19 IntervalFlush", 2024-04-16T07:46:08.794 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-04-16T07:46:08.794 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.795 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:08.795 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.795 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.795 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.795 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T07:46:08.795 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.748142+0000" 2024-04-16T07:46:08.795 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.795 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.795 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T07:46:08.795 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.795 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.795 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.795 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T07:46:08.795 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.753177+0000" 2024-04-16T07:46:08.795 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.796 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.796 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:08.796 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.796 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.796 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-16T07:46:08.796 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.796 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.796 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.796 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.796 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "peering_event", 2024-04-16T07:46:08.796 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 41, 2024-04-16T07:46:08.796 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.796 INFO:teuthology.orchestra.run.smithi070.stdout: "PeeringEvent": { 2024-04-16T07:46:08.796 INFO:teuthology.orchestra.run.smithi070.stdout: "from": "0", 2024-04-16T07:46:08.796 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "3.9", 2024-04-16T07:46:08.796 INFO:teuthology.orchestra.run.smithi070.stdout: "sent": 19, 2024-04-16T07:46:08.797 INFO:teuthology.orchestra.run.smithi070.stdout: "requested": 19, 2024-04-16T07:46:08.797 INFO:teuthology.orchestra.run.smithi070.stdout: "evt": "epoch_sent: 19 epoch_requested: 19 IntervalFlush", 2024-04-16T07:46:08.797 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-04-16T07:46:08.797 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.797 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:08.797 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.797 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.797 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.797 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T07:46:08.797 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.752071+0000" 2024-04-16T07:46:08.797 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.797 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.797 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T07:46:08.797 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.797 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.797 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.798 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T07:46:08.798 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.760823+0000" 2024-04-16T07:46:08.798 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.798 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.798 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:08.798 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.798 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.798 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-16T07:46:08.798 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.798 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.798 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.798 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.798 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "peering_event", 2024-04-16T07:46:08.798 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 42, 2024-04-16T07:46:08.799 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.799 INFO:teuthology.orchestra.run.smithi070.stdout: "PeeringEvent": { 2024-04-16T07:46:08.799 INFO:teuthology.orchestra.run.smithi070.stdout: "from": "0", 2024-04-16T07:46:08.799 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "3.f", 2024-04-16T07:46:08.799 INFO:teuthology.orchestra.run.smithi070.stdout: "sent": 19, 2024-04-16T07:46:08.799 INFO:teuthology.orchestra.run.smithi070.stdout: "requested": 19, 2024-04-16T07:46:08.799 INFO:teuthology.orchestra.run.smithi070.stdout: "evt": "epoch_sent: 19 epoch_requested: 19 IntervalFlush", 2024-04-16T07:46:08.799 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-04-16T07:46:08.799 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.799 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:08.799 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.799 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.799 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.799 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T07:46:08.799 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.748373+0000" 2024-04-16T07:46:08.799 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.800 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.800 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T07:46:08.800 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.800 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.800 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.800 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T07:46:08.800 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.752238+0000" 2024-04-16T07:46:08.800 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.800 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.800 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:08.800 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.800 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.800 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-16T07:46:08.800 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.800 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.800 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.801 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.801 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "peering_event", 2024-04-16T07:46:08.801 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 16777247, 2024-04-16T07:46:08.801 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.801 INFO:teuthology.orchestra.run.smithi070.stdout: "PeeringEvent": { 2024-04-16T07:46:08.801 INFO:teuthology.orchestra.run.smithi070.stdout: "from": "0", 2024-04-16T07:46:08.801 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "2.7", 2024-04-16T07:46:08.801 INFO:teuthology.orchestra.run.smithi070.stdout: "sent": 19, 2024-04-16T07:46:08.801 INFO:teuthology.orchestra.run.smithi070.stdout: "requested": 19, 2024-04-16T07:46:08.801 INFO:teuthology.orchestra.run.smithi070.stdout: "evt": "epoch_sent: 19 epoch_requested: 19 IntervalFlush", 2024-04-16T07:46:08.801 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-04-16T07:46:08.801 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.801 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:08.801 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.801 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.802 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.802 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T07:46:08.802 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.769916+0000" 2024-04-16T07:46:08.802 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.802 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.802 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T07:46:08.802 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.802 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.802 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.802 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T07:46:08.802 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.802 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.802 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.802 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:08.802 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.803 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.803 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-16T07:46:08.803 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.803 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.803 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.803 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.803 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "peering_event", 2024-04-16T07:46:08.803 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 16777248, 2024-04-16T07:46:08.803 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.803 INFO:teuthology.orchestra.run.smithi070.stdout: "PeeringEvent": { 2024-04-16T07:46:08.803 INFO:teuthology.orchestra.run.smithi070.stdout: "from": "0", 2024-04-16T07:46:08.803 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "3.7", 2024-04-16T07:46:08.803 INFO:teuthology.orchestra.run.smithi070.stdout: "sent": 19, 2024-04-16T07:46:08.803 INFO:teuthology.orchestra.run.smithi070.stdout: "requested": 19, 2024-04-16T07:46:08.803 INFO:teuthology.orchestra.run.smithi070.stdout: "evt": "epoch_sent: 19 epoch_requested: 19 IntervalFlush", 2024-04-16T07:46:08.804 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-04-16T07:46:08.804 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.804 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:08.804 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.804 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.804 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.804 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T07:46:08.804 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.770179+0000", 2024-04-16T07:46:08.804 INFO:teuthology.orchestra.run.smithi070.stdout: "blocker": { 2024-04-16T07:46:08.804 INFO:teuthology.orchestra.run.smithi070.stdout: "op_type": "PeeringEvent::PGPipeline::await_map", 2024-04-16T07:46:08.804 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.804 INFO:teuthology.orchestra.run.smithi070.stdout: "waiting": 1 2024-04-16T07:46:08.804 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.804 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.804 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.804 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.805 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T07:46:08.805 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.805 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.805 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.805 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T07:46:08.805 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.805 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.805 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.805 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:08.805 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.805 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.805 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-16T07:46:08.805 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.805 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.805 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.805 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.806 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "pg_advance_map", 2024-04-16T07:46:08.806 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 16777241, 2024-04-16T07:46:08.806 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.806 INFO:teuthology.orchestra.run.smithi070.stdout: "PGAdvanceMap": { 2024-04-16T07:46:08.806 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "2.0", 2024-04-16T07:46:08.806 INFO:teuthology.orchestra.run.smithi070.stdout: "from": 18, 2024-04-16T07:46:08.806 INFO:teuthology.orchestra.run.smithi070.stdout: "to": 19, 2024-04-16T07:46:08.806 INFO:teuthology.orchestra.run.smithi070.stdout: "do_init": false 2024-04-16T07:46:08.806 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.806 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.806 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.806 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.806 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "pg_advance_map", 2024-04-16T07:46:08.806 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 16777242, 2024-04-16T07:46:08.806 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.807 INFO:teuthology.orchestra.run.smithi070.stdout: "PGAdvanceMap": { 2024-04-16T07:46:08.807 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "2.2", 2024-04-16T07:46:08.807 INFO:teuthology.orchestra.run.smithi070.stdout: "from": 18, 2024-04-16T07:46:08.807 INFO:teuthology.orchestra.run.smithi070.stdout: "to": 19, 2024-04-16T07:46:08.807 INFO:teuthology.orchestra.run.smithi070.stdout: "do_init": false 2024-04-16T07:46:08.807 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.807 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.807 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.807 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.807 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "pg_advance_map", 2024-04-16T07:46:08.807 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 16777243, 2024-04-16T07:46:08.807 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.807 INFO:teuthology.orchestra.run.smithi070.stdout: "PGAdvanceMap": { 2024-04-16T07:46:08.807 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "2.7", 2024-04-16T07:46:08.807 INFO:teuthology.orchestra.run.smithi070.stdout: "from": 18, 2024-04-16T07:46:08.808 INFO:teuthology.orchestra.run.smithi070.stdout: "to": 19, 2024-04-16T07:46:08.808 INFO:teuthology.orchestra.run.smithi070.stdout: "do_init": false 2024-04-16T07:46:08.808 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.808 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.808 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.808 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.808 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "pg_advance_map", 2024-04-16T07:46:08.808 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 16777244, 2024-04-16T07:46:08.808 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.808 INFO:teuthology.orchestra.run.smithi070.stdout: "PGAdvanceMap": { 2024-04-16T07:46:08.808 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "3.7", 2024-04-16T07:46:08.808 INFO:teuthology.orchestra.run.smithi070.stdout: "from": 18, 2024-04-16T07:46:08.808 INFO:teuthology.orchestra.run.smithi070.stdout: "to": 19, 2024-04-16T07:46:08.808 INFO:teuthology.orchestra.run.smithi070.stdout: "do_init": false 2024-04-16T07:46:08.808 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.808 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.809 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.809 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.809 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "pg_advance_map", 2024-04-16T07:46:08.809 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 16777245, 2024-04-16T07:46:08.809 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.809 INFO:teuthology.orchestra.run.smithi070.stdout: "PGAdvanceMap": { 2024-04-16T07:46:08.809 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "3.c", 2024-04-16T07:46:08.809 INFO:teuthology.orchestra.run.smithi070.stdout: "from": 18, 2024-04-16T07:46:08.809 INFO:teuthology.orchestra.run.smithi070.stdout: "to": 19, 2024-04-16T07:46:08.809 INFO:teuthology.orchestra.run.smithi070.stdout: "do_init": false 2024-04-16T07:46:08.809 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.809 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.809 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.809 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.809 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "pg_advance_map", 2024-04-16T07:46:08.809 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 16777246, 2024-04-16T07:46:08.809 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.810 INFO:teuthology.orchestra.run.smithi070.stdout: "PGAdvanceMap": { 2024-04-16T07:46:08.810 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "3.d", 2024-04-16T07:46:08.810 INFO:teuthology.orchestra.run.smithi070.stdout: "from": 18, 2024-04-16T07:46:08.810 INFO:teuthology.orchestra.run.smithi070.stdout: "to": 19, 2024-04-16T07:46:08.810 INFO:teuthology.orchestra.run.smithi070.stdout: "do_init": false 2024-04-16T07:46:08.810 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.810 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.810 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.810 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.810 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "peering_event", 2024-04-16T07:46:08.810 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 33554463, 2024-04-16T07:46:08.810 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.810 INFO:teuthology.orchestra.run.smithi070.stdout: "PeeringEvent": { 2024-04-16T07:46:08.810 INFO:teuthology.orchestra.run.smithi070.stdout: "from": "0", 2024-04-16T07:46:08.811 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "3.b", 2024-04-16T07:46:08.811 INFO:teuthology.orchestra.run.smithi070.stdout: "sent": 19, 2024-04-16T07:46:08.811 INFO:teuthology.orchestra.run.smithi070.stdout: "requested": 19, 2024-04-16T07:46:08.811 INFO:teuthology.orchestra.run.smithi070.stdout: "evt": "epoch_sent: 19 epoch_requested: 19 IntervalFlush", 2024-04-16T07:46:08.811 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-04-16T07:46:08.811 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.811 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:08.811 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.811 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.811 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.811 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T07:46:08.811 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.758804+0000" 2024-04-16T07:46:08.811 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.811 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.812 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T07:46:08.812 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.812 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.812 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.812 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T07:46:08.812 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.759718+0000" 2024-04-16T07:46:08.812 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.812 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.812 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:08.812 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.812 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.812 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-16T07:46:08.812 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.812 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.812 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.812 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.813 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "peering_event", 2024-04-16T07:46:08.813 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 33554464, 2024-04-16T07:46:08.813 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.813 INFO:teuthology.orchestra.run.smithi070.stdout: "PeeringEvent": { 2024-04-16T07:46:08.813 INFO:teuthology.orchestra.run.smithi070.stdout: "from": "0", 2024-04-16T07:46:08.813 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "3.4", 2024-04-16T07:46:08.813 INFO:teuthology.orchestra.run.smithi070.stdout: "sent": 19, 2024-04-16T07:46:08.813 INFO:teuthology.orchestra.run.smithi070.stdout: "requested": 19, 2024-04-16T07:46:08.813 INFO:teuthology.orchestra.run.smithi070.stdout: "evt": "epoch_sent: 19 epoch_requested: 19 IntervalFlush", 2024-04-16T07:46:08.813 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-04-16T07:46:08.813 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.813 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:08.813 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.813 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.813 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.813 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T07:46:08.814 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.759673+0000" 2024-04-16T07:46:08.814 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.814 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.814 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T07:46:08.814 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.814 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.814 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.814 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T07:46:08.814 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.759952+0000", 2024-04-16T07:46:08.814 INFO:teuthology.orchestra.run.smithi070.stdout: "blocker": { 2024-04-16T07:46:08.814 INFO:teuthology.orchestra.run.smithi070.stdout: "op_type": "PeeringEvent::PGPipeline::process", 2024-04-16T07:46:08.814 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.814 INFO:teuthology.orchestra.run.smithi070.stdout: "waiting": 1 2024-04-16T07:46:08.815 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.815 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.815 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.815 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.815 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:08.815 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.815 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.815 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-16T07:46:08.815 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.815 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.815 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.815 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.815 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "peering_event", 2024-04-16T07:46:08.815 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 33554465, 2024-04-16T07:46:08.815 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.815 INFO:teuthology.orchestra.run.smithi070.stdout: "PeeringEvent": { 2024-04-16T07:46:08.816 INFO:teuthology.orchestra.run.smithi070.stdout: "from": "0", 2024-04-16T07:46:08.816 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "3.2", 2024-04-16T07:46:08.816 INFO:teuthology.orchestra.run.smithi070.stdout: "sent": 19, 2024-04-16T07:46:08.816 INFO:teuthology.orchestra.run.smithi070.stdout: "requested": 19, 2024-04-16T07:46:08.816 INFO:teuthology.orchestra.run.smithi070.stdout: "evt": "epoch_sent: 19 epoch_requested: 19 IntervalFlush", 2024-04-16T07:46:08.816 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-04-16T07:46:08.816 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.816 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:08.816 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.817 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.818 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.818 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T07:46:08.818 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.759424+0000" 2024-04-16T07:46:08.818 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.818 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.818 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T07:46:08.818 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.818 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.818 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.818 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T07:46:08.818 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.759848+0000" 2024-04-16T07:46:08.818 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.818 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.818 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:08.819 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.819 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.819 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-16T07:46:08.819 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.819 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.819 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.819 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.819 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "peering_event", 2024-04-16T07:46:08.819 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 33554466, 2024-04-16T07:46:08.819 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.819 INFO:teuthology.orchestra.run.smithi070.stdout: "PeeringEvent": { 2024-04-16T07:46:08.819 INFO:teuthology.orchestra.run.smithi070.stdout: "from": "0", 2024-04-16T07:46:08.819 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "3.a", 2024-04-16T07:46:08.819 INFO:teuthology.orchestra.run.smithi070.stdout: "sent": 19, 2024-04-16T07:46:08.819 INFO:teuthology.orchestra.run.smithi070.stdout: "requested": 19, 2024-04-16T07:46:08.819 INFO:teuthology.orchestra.run.smithi070.stdout: "evt": "epoch_sent: 19 epoch_requested: 19 IntervalFlush", 2024-04-16T07:46:08.819 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-04-16T07:46:08.820 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.820 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:08.820 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.820 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.820 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.820 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T07:46:08.820 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.760131+0000" 2024-04-16T07:46:08.820 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.820 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.820 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T07:46:08.820 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.820 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.820 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.820 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T07:46:08.820 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.760502+0000", 2024-04-16T07:46:08.820 INFO:teuthology.orchestra.run.smithi070.stdout: "blocker": { 2024-04-16T07:46:08.820 INFO:teuthology.orchestra.run.smithi070.stdout: "op_type": "PeeringEvent::PGPipeline::process", 2024-04-16T07:46:08.821 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.821 INFO:teuthology.orchestra.run.smithi070.stdout: "waiting": 1, 2024-04-16T07:46:08.821 INFO:teuthology.orchestra.run.smithi070.stdout: "held_by_operation_id": 33554461 2024-04-16T07:46:08.821 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.821 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.821 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.821 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.821 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:08.821 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.821 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.821 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-16T07:46:08.821 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.821 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.821 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.821 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.821 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "peering_event", 2024-04-16T07:46:08.821 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 33554467, 2024-04-16T07:46:08.821 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.822 INFO:teuthology.orchestra.run.smithi070.stdout: "PeeringEvent": { 2024-04-16T07:46:08.822 INFO:teuthology.orchestra.run.smithi070.stdout: "from": "0", 2024-04-16T07:46:08.822 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "2.5", 2024-04-16T07:46:08.822 INFO:teuthology.orchestra.run.smithi070.stdout: "sent": 19, 2024-04-16T07:46:08.822 INFO:teuthology.orchestra.run.smithi070.stdout: "requested": 19, 2024-04-16T07:46:08.822 INFO:teuthology.orchestra.run.smithi070.stdout: "evt": "epoch_sent: 19 epoch_requested: 19 IntervalFlush", 2024-04-16T07:46:08.822 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-04-16T07:46:08.822 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.822 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:08.822 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.822 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.822 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.822 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T07:46:08.822 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.760322+0000" 2024-04-16T07:46:08.822 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.822 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.823 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T07:46:08.823 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.823 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.823 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.823 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T07:46:08.823 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.760487+0000" 2024-04-16T07:46:08.823 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.823 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.823 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:08.823 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.823 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.823 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-16T07:46:08.823 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.823 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.823 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.823 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.823 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "peering_event", 2024-04-16T07:46:08.824 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 33554468, 2024-04-16T07:46:08.824 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.824 INFO:teuthology.orchestra.run.smithi070.stdout: "PeeringEvent": { 2024-04-16T07:46:08.824 INFO:teuthology.orchestra.run.smithi070.stdout: "from": "0", 2024-04-16T07:46:08.824 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "2.4", 2024-04-16T07:46:08.824 INFO:teuthology.orchestra.run.smithi070.stdout: "sent": 19, 2024-04-16T07:46:08.824 INFO:teuthology.orchestra.run.smithi070.stdout: "requested": 19, 2024-04-16T07:46:08.824 INFO:teuthology.orchestra.run.smithi070.stdout: "evt": "epoch_sent: 19 epoch_requested: 19 IntervalFlush", 2024-04-16T07:46:08.824 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-04-16T07:46:08.824 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.824 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:08.824 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.824 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.824 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.824 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T07:46:08.824 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.761088+0000" 2024-04-16T07:46:08.824 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.825 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.825 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T07:46:08.825 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.825 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.825 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.825 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T07:46:08.825 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-04-16T07:46:08.761149+0000", 2024-04-16T07:46:08.825 INFO:teuthology.orchestra.run.smithi070.stdout: "blocker": { 2024-04-16T07:46:08.825 INFO:teuthology.orchestra.run.smithi070.stdout: "op_type": "PeeringEvent::PGPipeline::process", 2024-04-16T07:46:08.825 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.825 INFO:teuthology.orchestra.run.smithi070.stdout: "waiting": 1, 2024-04-16T07:46:08.825 INFO:teuthology.orchestra.run.smithi070.stdout: "held_by_operation_id": 33554457 2024-04-16T07:46:08.825 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.825 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.825 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.825 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.825 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:08.826 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:08.826 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.826 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-16T07:46:08.826 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.826 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.826 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.826 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.826 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "pg_advance_map", 2024-04-16T07:46:08.826 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 33554457, 2024-04-16T07:46:08.826 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.826 INFO:teuthology.orchestra.run.smithi070.stdout: "PGAdvanceMap": { 2024-04-16T07:46:08.826 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "2.4", 2024-04-16T07:46:08.826 INFO:teuthology.orchestra.run.smithi070.stdout: "from": 18, 2024-04-16T07:46:08.826 INFO:teuthology.orchestra.run.smithi070.stdout: "to": 19, 2024-04-16T07:46:08.826 INFO:teuthology.orchestra.run.smithi070.stdout: "do_init": false 2024-04-16T07:46:08.826 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.827 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.827 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.827 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.827 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "pg_advance_map", 2024-04-16T07:46:08.827 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 33554460, 2024-04-16T07:46:08.827 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.827 INFO:teuthology.orchestra.run.smithi070.stdout: "PGAdvanceMap": { 2024-04-16T07:46:08.827 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "3.4", 2024-04-16T07:46:08.827 INFO:teuthology.orchestra.run.smithi070.stdout: "from": 18, 2024-04-16T07:46:08.827 INFO:teuthology.orchestra.run.smithi070.stdout: "to": 19, 2024-04-16T07:46:08.827 INFO:teuthology.orchestra.run.smithi070.stdout: "do_init": false 2024-04-16T07:46:08.827 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.827 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.827 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-16T07:46:08.827 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-16T07:46:08.827 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "pg_advance_map", 2024-04-16T07:46:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 33554461, 2024-04-16T07:46:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: "detail": { 2024-04-16T07:46:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: "PGAdvanceMap": { 2024-04-16T07:46:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: "pgid": "3.a", 2024-04-16T07:46:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: "from": 18, 2024-04-16T07:46:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: "to": 19, 2024-04-16T07:46:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: "do_init": false 2024-04-16T07:46:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-16T07:46:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-16T07:46:08.828 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-16T07:46:09.948 INFO:tasks.rados.rados.0.smithi070.stdout:73: finishing write tid 4 to smithi07031500-73 2024-04-16T07:46:09.951 INFO:tasks.rados.rados.0.smithi070.stdout:73: finishing write tid 5 to smithi07031500-73 2024-04-16T07:46:09.951 INFO:tasks.rados.rados.0.smithi070.stdout:73: oid 73 updating version 0 to 9 2024-04-16T07:46:09.951 INFO:tasks.rados.rados.0.smithi070.stdout:73: oid 73 version 9 is already newer than 0 2024-04-16T07:46:09.951 INFO:tasks.rados.rados.0.smithi070.stdout:73: oid 73 version 9 is already newer than 8 2024-04-16T07:46:09.951 INFO:tasks.rados.rados.0.smithi070.stdout:73: oid 73 version 9 is already newer than 7 2024-04-16T07:46:09.951 INFO:tasks.rados.rados.0.smithi070.stdout:update_object_version oid 73 v 9 (ObjNum 72 snap 0 seq_num 72) dirty exists 2024-04-16T07:46:09.951 INFO:tasks.rados.rados.0.smithi070.stderr:Error: racing read on 73 returned version 10 rather than version 9 2024-04-16T07:46:09.952 INFO:tasks.rados.rados.0.smithi070.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3109-g30b0aa00/rpm/el9/BUILD/ceph-19.0.0-3109-g30b0aa00/src/test/osd/RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)' thread 7f39c37fe640 time 2024-04-16T07:46:09.952648+0000 2024-04-16T07:46:09.952 INFO:tasks.rados.rados.0.smithi070.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3109-g30b0aa00/rpm/el9/BUILD/ceph-19.0.0-3109-g30b0aa00/src/test/osd/RadosModel.h: 1076: ceph_abort_msg("racing read got wrong version") 2024-04-16T07:46:09.952 INFO:tasks.rados.rados.0.smithi070.stderr: ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev) 2024-04-16T07:46:09.952 INFO:tasks.rados.rados.0.smithi070.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f39d004707a] 2024-04-16T07:46:09.952 INFO:tasks.rados.rados.0.smithi070.stderr: 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] 2024-04-16T07:46:09.952 INFO:tasks.rados.rados.0.smithi070.stderr: 3: (TestOp::finish(TestOp::CallbackInfo*)+0xe) [0x460e40] 2024-04-16T07:46:09.952 INFO:tasks.rados.rados.0.smithi070.stderr: 4: (write_callback(void*, void*)+0x1c) [0x460e7b] 2024-04-16T07:46:09.952 INFO:tasks.rados.rados.0.smithi070.stderr: 5: /lib64/librados.so.2(+0x95d9a) [0x7f39d0ab0d9a] 2024-04-16T07:46:09.952 INFO:tasks.rados.rados.0.smithi070.stderr: 6: /lib64/librados.so.2(+0x9fa94) [0x7f39d0abaa94] 2024-04-16T07:46:09.952 INFO:tasks.rados.rados.0.smithi070.stderr: 7: /lib64/librados.so.2(+0x94dc9) [0x7f39d0aafdc9] 2024-04-16T07:46:09.952 INFO:tasks.rados.rados.0.smithi070.stderr: 8: /lib64/librados.so.2(+0xa12c2) [0x7f39d0abc2c2] 2024-04-16T07:46:09.952 INFO:tasks.rados.rados.0.smithi070.stderr: 9: /lib64/librados.so.2(+0xa1bd2) [0x7f39d0abcbd2] 2024-04-16T07:46:09.952 INFO:tasks.rados.rados.0.smithi070.stderr: 10: /lib64/librados.so.2(+0xadc67) [0x7f39d0ac8c67] 2024-04-16T07:46:09.952 INFO:tasks.rados.rados.0.smithi070.stderr: 11: /lib64/librados.so.2(+0xae46d) [0x7f39d0ac946d] 2024-04-16T07:46:09.952 INFO:tasks.rados.rados.0.smithi070.stderr: 12: /lib64/librados.so.2(+0xb0a6f) [0x7f39d0acba6f] 2024-04-16T07:46:09.953 INFO:tasks.rados.rados.0.smithi070.stderr: 13: /lib64/librados.so.2(+0xb0ae8) [0x7f39d0acbae8] 2024-04-16T07:46:09.953 INFO:tasks.rados.rados.0.smithi070.stderr: 14: /lib64/librados.so.2(+0xb0b05) [0x7f39d0acbb05] 2024-04-16T07:46:09.953 INFO:tasks.rados.rados.0.smithi070.stderr: 15: /lib64/libstdc++.so.6(+0xdb924) [0x7f39d0cdb924] 2024-04-16T07:46:09.953 INFO:tasks.rados.rados.0.smithi070.stderr: 16: /lib64/libc.so.6(+0x9f802) [0x7f39cee9f802] 2024-04-16T07:46:09.953 INFO:tasks.rados.rados.0.smithi070.stderr: 17: /lib64/libc.so.6(+0x3f450) [0x7f39cee3f450] 2024-04-16T07:46:09.959 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:46:10.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:10.117+0000 7fd6cad5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:10.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:10.125+0000 7fd6cad5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:10.708 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T07:46:10.709 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/rados.py", line 289, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' 2024-04-16T07:46:10.710 ERROR:teuthology.run_tasks:Manager failed: rados Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/rados.py", line 308, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/rados.py", line 289, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' 2024-04-16T07:46:10.711 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-16T07:46:10.718 INFO:tasks.thrashosds:joining thrashosds 2024-04-16T07:46:13.830 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-16T07:46:13.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:13.953+0000 7f5e68624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:13.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:13.959+0000 7f5e68624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:13.973 INFO:teuthology.orchestra.run.smithi070.stdout:{} 2024-04-16T07:46:13.973 INFO:teuthology.orchestra.run.smithi070.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-16T07:46:13.984 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-16T07:46:14.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:14.092+0000 7f0d710ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:14.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:14.102+0000 7f0d710ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:14.690 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.1 to 1 (10000) 2024-04-16T07:46:14.708 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-04-16T07:46:14.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:14.818+0000 7f77be39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:14.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:14.828+0000 7f77be39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:15.695 INFO:teuthology.orchestra.run.smithi070.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-16T07:46:15.714 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-16T07:46:15.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:15.824+0000 7f3e356ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:15.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:15.834+0000 7f3e356ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:16.701 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.2 to 1 (10000) 2024-04-16T07:46:16.718 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-04-16T07:46:16.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:16.831+0000 7fda8466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:16.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:16.841+0000 7fda8466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:17.708 INFO:teuthology.orchestra.run.smithi070.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-16T07:46:17.725 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-16T07:46:17.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:17.856+0000 7f49f842d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:17.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:17.867+0000 7f49f842d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:18.709 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.3 to 1 (10000) 2024-04-16T07:46:18.725 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-04-16T07:46:18.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:18.838+0000 7fc39e4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:18.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:18.850+0000 7fc39e4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:19.714 INFO:teuthology.orchestra.run.smithi070.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-16T07:46:19.733 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-16T07:46:19.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:19.844+0000 7f8a49c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:19.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:19.854+0000 7f8a49c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:20.719 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.0 to 1 (10000) 2024-04-16T07:46:20.742 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-04-16T07:46:20.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:20.856+0000 7f00bad5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:20.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:20.866+0000 7f00bad5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:21.724 INFO:teuthology.orchestra.run.smithi070.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-16T07:46:21.740 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-16T07:46:21.740 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-16T07:46:21.740 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:46:21.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:21.852+0000 7f380786d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:21.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:21.859+0000 7f380786d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.048 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:46:22.048 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":28,"fsid":"e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6","created":"2024-04-16T07:45:37.265201+0000","modified":"2024-04-16T07:46:21.724325+0000","last_up_change":"2024-04-16T07:46:08.680981+0000","last_in_change":"2024-04-16T07:45:39.445562+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:45:45.641324+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T07:45:47.599173+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-04-16T07:45:53.412511+0000","flags":262305,"flags_names":"hashpspool,nopgchange,write_fadvise_dontneed,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1,"score_stable":1,"optimal_score":1,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"816d6f55-bf1d-48a9-9552-fc285b77358e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":19,"up_thru":19,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":4131536773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6801","nonce":4131536773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":4131536773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6803","nonce":4131536773}]},"public_addr":"172.21.15.70:6800/4131536773","cluster_addr":"172.21.15.70:6801/4131536773","heartbeat_back_addr":"172.21.15.70:6808/4131536773","heartbeat_front_addr":"172.21.15.70:6803/4131536773","state":["exists","up"]},{"osd":1,"uuid":"f777184a-794a-4817-ba31-872e243adcd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":873227738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6805","nonce":873227738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":873227738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6807","nonce":873227738}]},"public_addr":"172.21.15.70:6804/873227738","cluster_addr":"172.21.15.70:6805/873227738","heartbeat_back_addr":"172.21.15.70:6806/873227738","heartbeat_front_addr":"172.21.15.70:6807/873227738","state":["exists","up"]},{"osd":2,"uuid":"340d1b05-a919-45a7-bc75-1cb38943c8e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":911620790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":911620790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":911620790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":911620790}]},"public_addr":"172.21.15.172:6804/911620790","cluster_addr":"172.21.15.172:6805/911620790","heartbeat_back_addr":"172.21.15.172:6810/911620790","heartbeat_front_addr":"172.21.15.172:6809/911620790","state":["exists","up"]},{"osd":3,"uuid":"659133e9-2b60-43c8-9034-f4690c837fe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1749870805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6807","nonce":1749870805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":1749870805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6803","nonce":1749870805}]},"public_addr":"172.21.15.172:6808/1749870805","cluster_addr":"172.21.15.172:6807/1749870805","heartbeat_back_addr":"172.21.15.172:6806/1749870805","heartbeat_front_addr":"172.21.15.172:6803/1749870805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2024-04-16T07:46:07.678267+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:46:22.061 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-16T07:46:22.062 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:46:22.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:22.170+0000 7fadd782d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:22.178+0000 7fadd782d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.368 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:46:22.368 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":28,"fsid":"e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6","created":"2024-04-16T07:45:37.265201+0000","modified":"2024-04-16T07:46:21.724325+0000","last_up_change":"2024-04-16T07:46:08.680981+0000","last_in_change":"2024-04-16T07:45:39.445562+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:45:45.641324+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T07:45:47.599173+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-04-16T07:45:53.412511+0000","flags":262305,"flags_names":"hashpspool,nopgchange,write_fadvise_dontneed,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1,"score_stable":1,"optimal_score":1,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"816d6f55-bf1d-48a9-9552-fc285b77358e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":19,"up_thru":19,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":4131536773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6801","nonce":4131536773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":4131536773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6803","nonce":4131536773}]},"public_addr":"172.21.15.70:6800/4131536773","cluster_addr":"172.21.15.70:6801/4131536773","heartbeat_back_addr":"172.21.15.70:6808/4131536773","heartbeat_front_addr":"172.21.15.70:6803/4131536773","state":["exists","up"]},{"osd":1,"uuid":"f777184a-794a-4817-ba31-872e243adcd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":873227738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6805","nonce":873227738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":873227738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6807","nonce":873227738}]},"public_addr":"172.21.15.70:6804/873227738","cluster_addr":"172.21.15.70:6805/873227738","heartbeat_back_addr":"172.21.15.70:6806/873227738","heartbeat_front_addr":"172.21.15.70:6807/873227738","state":["exists","up"]},{"osd":2,"uuid":"340d1b05-a919-45a7-bc75-1cb38943c8e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":911620790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":911620790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":911620790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":911620790}]},"public_addr":"172.21.15.172:6804/911620790","cluster_addr":"172.21.15.172:6805/911620790","heartbeat_back_addr":"172.21.15.172:6810/911620790","heartbeat_front_addr":"172.21.15.172:6809/911620790","state":["exists","up"]},{"osd":3,"uuid":"659133e9-2b60-43c8-9034-f4690c837fe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1749870805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6807","nonce":1749870805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":1749870805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6803","nonce":1749870805}]},"public_addr":"172.21.15.172:6808/1749870805","cluster_addr":"172.21.15.172:6807/1749870805","heartbeat_back_addr":"172.21.15.172:6806/1749870805","heartbeat_front_addr":"172.21.15.172:6803/1749870805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2024-04-16T07:46:07.678267+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:46:22.381 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-16T07:46:22.381 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-16T07:46:22.382 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-16T07:46:22.382 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-16T07:46:22.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:22.500+0000 7f49af2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:22.500+0000 7f8013b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:22.500+0000 7f176b39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:22.500+0000 7fc31de6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:22.510+0000 7f176b39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:22.510+0000 7f49af2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:22.510+0000 7fc31de6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:22.510+0000 7f8013b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.542 INFO:teuthology.orchestra.run.smithi070.stdout:81604378637 2024-04-16T07:46:22.542 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T07:46:22.546 INFO:teuthology.orchestra.run.smithi070.stdout:38654705701 2024-04-16T07:46:22.546 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T07:46:22.547 INFO:teuthology.orchestra.run.smithi070.stdout:38654705701 2024-04-16T07:46:22.547 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-16T07:46:22.552 INFO:teuthology.orchestra.run.smithi070.stdout:38654705701 2024-04-16T07:46:22.552 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T07:46:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:22.658+0000 7f03a9f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:22.669+0000 7f03a9f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:22.706+0000 7fedf2c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:22.707+0000 7f92f26ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:22.716+0000 7fedf2c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:22.716+0000 7f92f26ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:22.763+0000 7ffbb806d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:22.772+0000 7ffbb806d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:22.874 INFO:teuthology.orchestra.run.smithi070.stdout:81604378633 2024-04-16T07:46:22.893 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378637 got 81604378633 for osd.0 2024-04-16T07:46:22.928 INFO:teuthology.orchestra.run.smithi070.stdout:38654705698 2024-04-16T07:46:22.940 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705701 got 38654705698 for osd.3 2024-04-16T07:46:22.944 INFO:teuthology.orchestra.run.smithi070.stdout:38654705698 2024-04-16T07:46:22.957 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705701 got 38654705698 for osd.2 2024-04-16T07:46:22.982 INFO:teuthology.orchestra.run.smithi070.stdout:38654705698 2024-04-16T07:46:22.995 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705701 got 38654705698 for osd.1 2024-04-16T07:46:23.894 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T07:46:23.941 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-16T07:46:23.958 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T07:46:23.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T07:46:24.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:24.010+0000 7f2f28017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:24.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:24.021+0000 7f2f28017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:24.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:24.058+0000 7fcda800f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:24.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:24.069+0000 7fcda800f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:24.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:24.103+0000 7f2fa1754640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:24.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:24.113+0000 7f2fa1754640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:24.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:24.140+0000 7f51f846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:24.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:24.149+0000 7f51f846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:24.221 INFO:teuthology.orchestra.run.smithi070.stdout:81604378638 2024-04-16T07:46:24.235 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378637 got 81604378638 for osd.0 2024-04-16T07:46:24.235 DEBUG:teuthology.parallel:result is None 2024-04-16T07:46:24.270 INFO:teuthology.orchestra.run.smithi070.stdout:38654705701 2024-04-16T07:46:24.283 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705701 got 38654705701 for osd.3 2024-04-16T07:46:24.283 DEBUG:teuthology.parallel:result is None 2024-04-16T07:46:24.321 INFO:teuthology.orchestra.run.smithi070.stdout:38654705701 2024-04-16T07:46:24.334 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705701 got 38654705701 for osd.2 2024-04-16T07:46:24.334 DEBUG:teuthology.parallel:result is None 2024-04-16T07:46:24.346 INFO:teuthology.orchestra.run.smithi070.stdout:38654705701 2024-04-16T07:46:24.359 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705701 got 38654705701 for osd.1 2024-04-16T07:46:24.359 DEBUG:teuthology.parallel:result is None 2024-04-16T07:46:24.359 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-16T07:46:24.359 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:46:24.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:24.508+0000 7fbca4ffa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:24.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:24.516+0000 7fbca4ffa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:24.704 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:46:24.704 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-16T07:46:24.717 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-04-16T07:46:23.627148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":213802021,"num_objects":81,"num_object_clones":0,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":258,"num_read_kb":167,"num_write":474,"num_write_kb":97659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2079924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":382,"ondisk_log_size":382,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":427128,"kb_used_data":319704,"kb_used_omap":0,"kb_used_meta":107392,"kb_avail":377060232,"statfs":{"total":386547056640,"available":386109677568,"internally_reserved":0,"allocated":327376896,"data_stored":325866073,"data_compressed":89727,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109969408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.916384"},"pg_stats":[{"pgid":"3.f","version":"17'19","reported_seq":72,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.742216+0000","last_change":"2024-04-16T07:46:09.858371+0000","last_active":"2024-04-16T07:46:21.742216+0000","last_peered":"2024-04-16T07:46:21.742216+0000","last_clean":"2024-04-16T07:46:21.742216+0000","last_became_active":"2024-04-16T07:46:09.847369+0000","last_became_peered":"2024-04-16T07:46:09.847369+0000","last_unstale":"2024-04-16T07:46:21.742216+0000","last_undegraded":"2024-04-16T07:46:21.742216+0000","last_fullsized":"2024-04-16T07:46:21.742216+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16537890,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":25,"num_write_kb":7533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"17'27","reported_seq":71,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.729138+0000","last_change":"2024-04-16T07:45:54.897183+0000","last_active":"2024-04-16T07:46:19.729138+0000","last_peered":"2024-04-16T07:46:19.729138+0000","last_clean":"2024-04-16T07:46:19.729138+0000","last_became_active":"2024-04-16T07:45:54.891021+0000","last_became_peered":"2024-04-16T07:45:54.891021+0000","last_unstale":"2024-04-16T07:46:19.729138+0000","last_undegraded":"2024-04-16T07:46:19.729138+0000","last_fullsized":"2024-04-16T07:46:19.729138+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20353013,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":8,"num_write":35,"num_write_kb":8984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'20","reported_seq":67,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.768372+0000","last_change":"2024-04-16T07:46:10.109586+0000","last_active":"2024-04-16T07:46:20.768372+0000","last_peered":"2024-04-16T07:46:20.768372+0000","last_clean":"2024-04-16T07:46:20.768372+0000","last_became_active":"2024-04-16T07:46:09.820109+0000","last_became_peered":"2024-04-16T07:46:09.820109+0000","last_unstale":"2024-04-16T07:46:20.768372+0000","last_undegraded":"2024-04-16T07:46:20.768372+0000","last_fullsized":"2024-04-16T07:46:20.768372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16067294,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":26,"num_write_kb":7323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'32","reported_seq":89,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.754522+0000","last_change":"2024-04-16T07:46:09.896432+0000","last_active":"2024-04-16T07:46:21.754522+0000","last_peered":"2024-04-16T07:46:21.754522+0000","last_clean":"2024-04-16T07:46:21.754522+0000","last_became_active":"2024-04-16T07:46:09.891795+0000","last_became_peered":"2024-04-16T07:46:09.891795+0000","last_unstale":"2024-04-16T07:46:21.754522+0000","last_undegraded":"2024-04-16T07:46:21.754522+0000","last_fullsized":"2024-04-16T07:46:21.754522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25159749,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":9,"num_write":42,"num_write_kb":11798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'12","reported_seq":54,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.834431+0000","last_change":"2024-04-16T07:46:09.811202+0000","last_active":"2024-04-16T07:46:18.834431+0000","last_peered":"2024-04-16T07:46:18.834431+0000","last_clean":"2024-04-16T07:46:18.834431+0000","last_became_active":"2024-04-16T07:46:09.806014+0000","last_became_peered":"2024-04-16T07:46:09.806014+0000","last_unstale":"2024-04-16T07:46:18.834431+0000","last_undegraded":"2024-04-16T07:46:18.834431+0000","last_fullsized":"2024-04-16T07:46:18.834431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'5","reported_seq":54,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.753883+0000","last_change":"2024-04-16T07:46:09.873821+0000","last_active":"2024-04-16T07:46:21.753883+0000","last_peered":"2024-04-16T07:46:21.753883+0000","last_clean":"2024-04-16T07:46:21.753883+0000","last_became_active":"2024-04-16T07:46:09.869336+0000","last_became_peered":"2024-04-16T07:46:09.869336+0000","last_unstale":"2024-04-16T07:46:21.753883+0000","last_undegraded":"2024-04-16T07:46:21.753883+0000","last_fullsized":"2024-04-16T07:46:21.753883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4269570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":1976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'26","reported_seq":85,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.741032+0000","last_change":"2024-04-16T07:46:10.147324+0000","last_active":"2024-04-16T07:46:21.741032+0000","last_peered":"2024-04-16T07:46:21.741032+0000","last_clean":"2024-04-16T07:46:21.741032+0000","last_became_active":"2024-04-16T07:46:09.841067+0000","last_became_peered":"2024-04-16T07:46:09.841067+0000","last_unstale":"2024-04-16T07:46:21.741032+0000","last_undegraded":"2024-04-16T07:46:21.741032+0000","last_fullsized":"2024-04-16T07:46:21.741032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"17'23","reported_seq":65,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.759131+0000","last_change":"2024-04-16T07:45:54.797160+0000","last_active":"2024-04-16T07:46:20.759131+0000","last_peered":"2024-04-16T07:46:20.759131+0000","last_clean":"2024-04-16T07:46:20.759131+0000","last_became_active":"2024-04-16T07:45:54.786910+0000","last_became_peered":"2024-04-16T07:45:54.786910+0000","last_unstale":"2024-04-16T07:46:20.759131+0000","last_undegraded":"2024-04-16T07:46:20.759131+0000","last_fullsized":"2024-04-16T07:46:20.759131+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19294205,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":30,"num_write_kb":8200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":175,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.817843+0000","last_change":"2024-04-16T07:46:09.797708+0000","last_active":"2024-04-16T07:46:18.817843+0000","last_peered":"2024-04-16T07:46:18.817843+0000","last_clean":"2024-04-16T07:46:18.817843+0000","last_became_active":"2024-04-16T07:46:09.792221+0000","last_became_peered":"2024-04-16T07:46:09.792221+0000","last_unstale":"2024-04-16T07:46:18.817843+0000","last_undegraded":"2024-04-16T07:46:18.817843+0000","last_fullsized":"2024-04-16T07:46:18.817843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_clean_scrub_stamp":"2024-04-16T07:45:46.626705+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":45,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.772073+0000","last_change":"2024-04-16T07:46:09.796645+0000","last_active":"2024-04-16T07:46:20.772073+0000","last_peered":"2024-04-16T07:46:20.772073+0000","last_clean":"2024-04-16T07:46:20.772073+0000","last_became_active":"2024-04-16T07:46:09.790535+0000","last_became_peered":"2024-04-16T07:46:09.790535+0000","last_unstale":"2024-04-16T07:46:20.772073+0000","last_undegraded":"2024-04-16T07:46:20.772073+0000","last_fullsized":"2024-04-16T07:46:20.772073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"20'10","reported_seq":54,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.766709+0000","last_change":"2024-04-16T07:46:10.004761+0000","last_active":"2024-04-16T07:46:20.766709+0000","last_peered":"2024-04-16T07:46:20.766709+0000","last_clean":"2024-04-16T07:46:20.766709+0000","last_became_active":"2024-04-16T07:46:09.806452+0000","last_became_peered":"2024-04-16T07:46:09.806452+0000","last_unstale":"2024-04-16T07:46:20.766709+0000","last_undegraded":"2024-04-16T07:46:20.766709+0000","last_fullsized":"2024-04-16T07:46:20.766709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8184910,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":13,"num_write_kb":3711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":626546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":42,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.745900+0000","last_change":"2024-04-16T07:46:09.913777+0000","last_active":"2024-04-16T07:46:21.745900+0000","last_peered":"2024-04-16T07:46:21.745900+0000","last_clean":"2024-04-16T07:46:21.745900+0000","last_became_active":"2024-04-16T07:46:09.909374+0000","last_became_peered":"2024-04-16T07:46:09.909374+0000","last_unstale":"2024-04-16T07:46:21.745900+0000","last_undegraded":"2024-04-16T07:46:21.745900+0000","last_fullsized":"2024-04-16T07:46:21.745900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"20'14","reported_seq":61,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.829023+0000","last_change":"2024-04-16T07:46:10.087832+0000","last_active":"2024-04-16T07:46:18.829023+0000","last_peered":"2024-04-16T07:46:18.829023+0000","last_clean":"2024-04-16T07:46:18.829023+0000","last_became_active":"2024-04-16T07:46:09.828384+0000","last_became_peered":"2024-04-16T07:46:09.828384+0000","last_unstale":"2024-04-16T07:46:18.829023+0000","last_undegraded":"2024-04-16T07:46:18.829023+0000","last_fullsized":"2024-04-16T07:46:18.829023+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11462242,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":18,"num_write_kb":5591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":481348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":44,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.755443+0000","last_change":"2024-04-16T07:45:48.724954+0000","last_active":"2024-04-16T07:46:20.755443+0000","last_peered":"2024-04-16T07:46:20.755443+0000","last_clean":"2024-04-16T07:46:20.755443+0000","last_became_active":"2024-04-16T07:45:48.719985+0000","last_became_peered":"2024-04-16T07:45:48.719985+0000","last_unstale":"2024-04-16T07:46:20.755443+0000","last_undegraded":"2024-04-16T07:46:20.755443+0000","last_fullsized":"2024-04-16T07:46:20.755443+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'6","reported_seq":45,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.826499+0000","last_change":"2024-04-16T07:45:54.815123+0000","last_active":"2024-04-16T07:46:18.826499+0000","last_peered":"2024-04-16T07:46:18.826499+0000","last_clean":"2024-04-16T07:46:18.826499+0000","last_became_active":"2024-04-16T07:45:54.810589+0000","last_became_peered":"2024-04-16T07:45:54.810589+0000","last_unstale":"2024-04-16T07:46:18.826499+0000","last_undegraded":"2024-04-16T07:46:18.826499+0000","last_fullsized":"2024-04-16T07:46:18.826499+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5327753,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":42,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.755496+0000","last_change":"2024-04-16T07:46:09.905947+0000","last_active":"2024-04-16T07:46:21.755496+0000","last_peered":"2024-04-16T07:46:21.755496+0000","last_clean":"2024-04-16T07:46:21.755496+0000","last_became_active":"2024-04-16T07:46:09.900754+0000","last_became_peered":"2024-04-16T07:46:09.900754+0000","last_unstale":"2024-04-16T07:46:21.755496+0000","last_undegraded":"2024-04-16T07:46:21.755496+0000","last_fullsized":"2024-04-16T07:46:21.755496+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"20'24","reported_seq":73,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.733418+0000","last_change":"2024-04-16T07:46:10.052676+0000","last_active":"2024-04-16T07:46:19.733418+0000","last_peered":"2024-04-16T07:46:19.733418+0000","last_clean":"2024-04-16T07:46:19.733418+0000","last_became_active":"2024-04-16T07:46:09.837421+0000","last_became_peered":"2024-04-16T07:46:09.837421+0000","last_unstale":"2024-04-16T07:46:19.733418+0000","last_undegraded":"2024-04-16T07:46:19.733418+0000","last_fullsized":"2024-04-16T07:46:19.733418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20991712,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":31,"num_write_kb":9471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":930388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'18","reported_seq":58,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.750437+0000","last_change":"2024-04-16T07:45:54.812310+0000","last_active":"2024-04-16T07:46:20.750437+0000","last_peered":"2024-04-16T07:46:20.750437+0000","last_clean":"2024-04-16T07:46:20.750437+0000","last_became_active":"2024-04-16T07:45:54.804593+0000","last_became_peered":"2024-04-16T07:45:54.804593+0000","last_unstale":"2024-04-16T07:46:20.750437+0000","last_undegraded":"2024-04-16T07:46:20.750437+0000","last_fullsized":"2024-04-16T07:46:20.750437+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14571504,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":23,"num_write_kb":6959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":44,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.743780+0000","last_change":"2024-04-16T07:46:09.886637+0000","last_active":"2024-04-16T07:46:19.743780+0000","last_peered":"2024-04-16T07:46:19.743780+0000","last_clean":"2024-04-16T07:46:19.743780+0000","last_became_active":"2024-04-16T07:46:09.882028+0000","last_became_peered":"2024-04-16T07:46:09.882028+0000","last_unstale":"2024-04-16T07:46:19.743780+0000","last_undegraded":"2024-04-16T07:46:19.743780+0000","last_fullsized":"2024-04-16T07:46:19.743780+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"20'20","reported_seq":68,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.734207+0000","last_change":"2024-04-16T07:46:09.984076+0000","last_active":"2024-04-16T07:46:19.734207+0000","last_peered":"2024-04-16T07:46:19.734207+0000","last_clean":"2024-04-16T07:46:19.734207+0000","last_became_active":"2024-04-16T07:46:09.828458+0000","last_became_peered":"2024-04-16T07:46:09.828458+0000","last_unstale":"2024-04-16T07:46:19.734207+0000","last_undegraded":"2024-04-16T07:46:19.734207+0000","last_fullsized":"2024-04-16T07:46:19.734207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17563117,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":26,"num_write_kb":7290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":41642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":45,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.836114+0000","last_change":"2024-04-16T07:46:09.857758+0000","last_active":"2024-04-16T07:46:18.836114+0000","last_peered":"2024-04-16T07:46:18.836114+0000","last_clean":"2024-04-16T07:46:18.836114+0000","last_became_active":"2024-04-16T07:46:09.850258+0000","last_became_peered":"2024-04-16T07:46:09.850258+0000","last_unstale":"2024-04-16T07:46:18.836114+0000","last_undegraded":"2024-04-16T07:46:18.836114+0000","last_fullsized":"2024-04-16T07:46:18.836114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'10","reported_seq":56,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.835308+0000","last_change":"2024-04-16T07:46:09.975356+0000","last_active":"2024-04-16T07:46:18.835308+0000","last_peered":"2024-04-16T07:46:18.835308+0000","last_clean":"2024-04-16T07:46:18.835308+0000","last_became_active":"2024-04-16T07:46:09.802912+0000","last_became_peered":"2024-04-16T07:46:09.802912+0000","last_unstale":"2024-04-16T07:46:18.835308+0000","last_undegraded":"2024-04-16T07:46:18.835308+0000","last_fullsized":"2024-04-16T07:46:18.835308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7428595,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":13,"num_write_kb":3079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":44,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.740283+0000","last_change":"2024-04-16T07:46:09.906633+0000","last_active":"2024-04-16T07:46:19.740283+0000","last_peered":"2024-04-16T07:46:19.740283+0000","last_clean":"2024-04-16T07:46:19.740283+0000","last_became_active":"2024-04-16T07:46:09.896255+0000","last_became_peered":"2024-04-16T07:46:09.896255+0000","last_unstale":"2024-04-16T07:46:19.740283+0000","last_undegraded":"2024-04-16T07:46:19.740283+0000","last_fullsized":"2024-04-16T07:46:19.740283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'19","reported_seq":61,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.729909+0000","last_change":"2024-04-16T07:45:54.819481+0000","last_active":"2024-04-16T07:46:19.729909+0000","last_peered":"2024-04-16T07:46:19.729909+0000","last_clean":"2024-04-16T07:46:19.729909+0000","last_became_active":"2024-04-16T07:45:54.813131+0000","last_became_peered":"2024-04-16T07:45:54.813131+0000","last_unstale":"2024-04-16T07:46:19.729909+0000","last_undegraded":"2024-04-16T07:46:19.729909+0000","last_fullsized":"2024-04-16T07:46:19.729909+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16184943,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":25,"num_write_kb":6850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":45,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.769122+0000","last_change":"2024-04-16T07:46:09.836581+0000","last_active":"2024-04-16T07:46:20.769122+0000","last_peered":"2024-04-16T07:46:20.769122+0000","last_clean":"2024-04-16T07:46:20.769122+0000","last_became_active":"2024-04-16T07:46:09.830316+0000","last_became_peered":"2024-04-16T07:46:09.830316+0000","last_unstale":"2024-04-16T07:46:20.769122+0000","last_undegraded":"2024-04-16T07:46:20.769122+0000","last_fullsized":"2024-04-16T07:46:20.769122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":213211653,"num_objects":79,"num_object_clones":0,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":155,"num_read_kb":76,"num_write":337,"num_write_kb":95765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2079924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":285,"ondisk_log_size":285,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705701,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":121520,"kb_used_data":94696,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94250320,"statfs":{"total":96636764160,"available":96512327680,"internally_reserved":0,"allocated":96968704,"data_stored":96608010,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":105888,"kb_used_data":79064,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94265952,"statfs":{"total":96636764160,"available":96528334848,"internally_reserved":0,"allocated":80961536,"data_stored":80730251,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705701,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99952,"kb_used_data":73128,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94271888,"statfs":{"total":96636764160,"available":96534413312,"internally_reserved":0,"allocated":74883072,"data_stored":74143430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378638,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99768,"kb_used_data":72816,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94272072,"statfs":{"total":96636764160,"available":96534601728,"internally_reserved":0,"allocated":74563584,"data_stored":74384382,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:46:24.718 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:46:24.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:24.837+0000 7f03f8cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:24.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:24.845+0000 7f03f8cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:25.034 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:46:25.034 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-16T07:46:25.046 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-04-16T07:46:23.627148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":213802021,"num_objects":81,"num_object_clones":0,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":258,"num_read_kb":167,"num_write":474,"num_write_kb":97659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2079924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":382,"ondisk_log_size":382,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":427128,"kb_used_data":319704,"kb_used_omap":0,"kb_used_meta":107392,"kb_avail":377060232,"statfs":{"total":386547056640,"available":386109677568,"internally_reserved":0,"allocated":327376896,"data_stored":325866073,"data_compressed":89727,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109969408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.916384"},"pg_stats":[{"pgid":"3.f","version":"17'19","reported_seq":72,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.742216+0000","last_change":"2024-04-16T07:46:09.858371+0000","last_active":"2024-04-16T07:46:21.742216+0000","last_peered":"2024-04-16T07:46:21.742216+0000","last_clean":"2024-04-16T07:46:21.742216+0000","last_became_active":"2024-04-16T07:46:09.847369+0000","last_became_peered":"2024-04-16T07:46:09.847369+0000","last_unstale":"2024-04-16T07:46:21.742216+0000","last_undegraded":"2024-04-16T07:46:21.742216+0000","last_fullsized":"2024-04-16T07:46:21.742216+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16537890,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":25,"num_write_kb":7533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"17'27","reported_seq":71,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.729138+0000","last_change":"2024-04-16T07:45:54.897183+0000","last_active":"2024-04-16T07:46:19.729138+0000","last_peered":"2024-04-16T07:46:19.729138+0000","last_clean":"2024-04-16T07:46:19.729138+0000","last_became_active":"2024-04-16T07:45:54.891021+0000","last_became_peered":"2024-04-16T07:45:54.891021+0000","last_unstale":"2024-04-16T07:46:19.729138+0000","last_undegraded":"2024-04-16T07:46:19.729138+0000","last_fullsized":"2024-04-16T07:46:19.729138+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20353013,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":8,"num_write":35,"num_write_kb":8984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'20","reported_seq":67,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.768372+0000","last_change":"2024-04-16T07:46:10.109586+0000","last_active":"2024-04-16T07:46:20.768372+0000","last_peered":"2024-04-16T07:46:20.768372+0000","last_clean":"2024-04-16T07:46:20.768372+0000","last_became_active":"2024-04-16T07:46:09.820109+0000","last_became_peered":"2024-04-16T07:46:09.820109+0000","last_unstale":"2024-04-16T07:46:20.768372+0000","last_undegraded":"2024-04-16T07:46:20.768372+0000","last_fullsized":"2024-04-16T07:46:20.768372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16067294,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":26,"num_write_kb":7323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'32","reported_seq":89,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.754522+0000","last_change":"2024-04-16T07:46:09.896432+0000","last_active":"2024-04-16T07:46:21.754522+0000","last_peered":"2024-04-16T07:46:21.754522+0000","last_clean":"2024-04-16T07:46:21.754522+0000","last_became_active":"2024-04-16T07:46:09.891795+0000","last_became_peered":"2024-04-16T07:46:09.891795+0000","last_unstale":"2024-04-16T07:46:21.754522+0000","last_undegraded":"2024-04-16T07:46:21.754522+0000","last_fullsized":"2024-04-16T07:46:21.754522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25159749,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":9,"num_write":42,"num_write_kb":11798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'12","reported_seq":54,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.834431+0000","last_change":"2024-04-16T07:46:09.811202+0000","last_active":"2024-04-16T07:46:18.834431+0000","last_peered":"2024-04-16T07:46:18.834431+0000","last_clean":"2024-04-16T07:46:18.834431+0000","last_became_active":"2024-04-16T07:46:09.806014+0000","last_became_peered":"2024-04-16T07:46:09.806014+0000","last_unstale":"2024-04-16T07:46:18.834431+0000","last_undegraded":"2024-04-16T07:46:18.834431+0000","last_fullsized":"2024-04-16T07:46:18.834431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'5","reported_seq":54,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.753883+0000","last_change":"2024-04-16T07:46:09.873821+0000","last_active":"2024-04-16T07:46:21.753883+0000","last_peered":"2024-04-16T07:46:21.753883+0000","last_clean":"2024-04-16T07:46:21.753883+0000","last_became_active":"2024-04-16T07:46:09.869336+0000","last_became_peered":"2024-04-16T07:46:09.869336+0000","last_unstale":"2024-04-16T07:46:21.753883+0000","last_undegraded":"2024-04-16T07:46:21.753883+0000","last_fullsized":"2024-04-16T07:46:21.753883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4269570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":1976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'26","reported_seq":85,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.741032+0000","last_change":"2024-04-16T07:46:10.147324+0000","last_active":"2024-04-16T07:46:21.741032+0000","last_peered":"2024-04-16T07:46:21.741032+0000","last_clean":"2024-04-16T07:46:21.741032+0000","last_became_active":"2024-04-16T07:46:09.841067+0000","last_became_peered":"2024-04-16T07:46:09.841067+0000","last_unstale":"2024-04-16T07:46:21.741032+0000","last_undegraded":"2024-04-16T07:46:21.741032+0000","last_fullsized":"2024-04-16T07:46:21.741032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"17'23","reported_seq":65,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.759131+0000","last_change":"2024-04-16T07:45:54.797160+0000","last_active":"2024-04-16T07:46:20.759131+0000","last_peered":"2024-04-16T07:46:20.759131+0000","last_clean":"2024-04-16T07:46:20.759131+0000","last_became_active":"2024-04-16T07:45:54.786910+0000","last_became_peered":"2024-04-16T07:45:54.786910+0000","last_unstale":"2024-04-16T07:46:20.759131+0000","last_undegraded":"2024-04-16T07:46:20.759131+0000","last_fullsized":"2024-04-16T07:46:20.759131+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19294205,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":30,"num_write_kb":8200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":175,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.817843+0000","last_change":"2024-04-16T07:46:09.797708+0000","last_active":"2024-04-16T07:46:18.817843+0000","last_peered":"2024-04-16T07:46:18.817843+0000","last_clean":"2024-04-16T07:46:18.817843+0000","last_became_active":"2024-04-16T07:46:09.792221+0000","last_became_peered":"2024-04-16T07:46:09.792221+0000","last_unstale":"2024-04-16T07:46:18.817843+0000","last_undegraded":"2024-04-16T07:46:18.817843+0000","last_fullsized":"2024-04-16T07:46:18.817843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_clean_scrub_stamp":"2024-04-16T07:45:46.626705+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":45,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.772073+0000","last_change":"2024-04-16T07:46:09.796645+0000","last_active":"2024-04-16T07:46:20.772073+0000","last_peered":"2024-04-16T07:46:20.772073+0000","last_clean":"2024-04-16T07:46:20.772073+0000","last_became_active":"2024-04-16T07:46:09.790535+0000","last_became_peered":"2024-04-16T07:46:09.790535+0000","last_unstale":"2024-04-16T07:46:20.772073+0000","last_undegraded":"2024-04-16T07:46:20.772073+0000","last_fullsized":"2024-04-16T07:46:20.772073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"20'10","reported_seq":54,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.766709+0000","last_change":"2024-04-16T07:46:10.004761+0000","last_active":"2024-04-16T07:46:20.766709+0000","last_peered":"2024-04-16T07:46:20.766709+0000","last_clean":"2024-04-16T07:46:20.766709+0000","last_became_active":"2024-04-16T07:46:09.806452+0000","last_became_peered":"2024-04-16T07:46:09.806452+0000","last_unstale":"2024-04-16T07:46:20.766709+0000","last_undegraded":"2024-04-16T07:46:20.766709+0000","last_fullsized":"2024-04-16T07:46:20.766709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8184910,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":13,"num_write_kb":3711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":626546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":42,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.745900+0000","last_change":"2024-04-16T07:46:09.913777+0000","last_active":"2024-04-16T07:46:21.745900+0000","last_peered":"2024-04-16T07:46:21.745900+0000","last_clean":"2024-04-16T07:46:21.745900+0000","last_became_active":"2024-04-16T07:46:09.909374+0000","last_became_peered":"2024-04-16T07:46:09.909374+0000","last_unstale":"2024-04-16T07:46:21.745900+0000","last_undegraded":"2024-04-16T07:46:21.745900+0000","last_fullsized":"2024-04-16T07:46:21.745900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"20'14","reported_seq":61,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.829023+0000","last_change":"2024-04-16T07:46:10.087832+0000","last_active":"2024-04-16T07:46:18.829023+0000","last_peered":"2024-04-16T07:46:18.829023+0000","last_clean":"2024-04-16T07:46:18.829023+0000","last_became_active":"2024-04-16T07:46:09.828384+0000","last_became_peered":"2024-04-16T07:46:09.828384+0000","last_unstale":"2024-04-16T07:46:18.829023+0000","last_undegraded":"2024-04-16T07:46:18.829023+0000","last_fullsized":"2024-04-16T07:46:18.829023+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11462242,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":18,"num_write_kb":5591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":481348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":44,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.755443+0000","last_change":"2024-04-16T07:45:48.724954+0000","last_active":"2024-04-16T07:46:20.755443+0000","last_peered":"2024-04-16T07:46:20.755443+0000","last_clean":"2024-04-16T07:46:20.755443+0000","last_became_active":"2024-04-16T07:45:48.719985+0000","last_became_peered":"2024-04-16T07:45:48.719985+0000","last_unstale":"2024-04-16T07:46:20.755443+0000","last_undegraded":"2024-04-16T07:46:20.755443+0000","last_fullsized":"2024-04-16T07:46:20.755443+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'6","reported_seq":45,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.826499+0000","last_change":"2024-04-16T07:45:54.815123+0000","last_active":"2024-04-16T07:46:18.826499+0000","last_peered":"2024-04-16T07:46:18.826499+0000","last_clean":"2024-04-16T07:46:18.826499+0000","last_became_active":"2024-04-16T07:45:54.810589+0000","last_became_peered":"2024-04-16T07:45:54.810589+0000","last_unstale":"2024-04-16T07:46:18.826499+0000","last_undegraded":"2024-04-16T07:46:18.826499+0000","last_fullsized":"2024-04-16T07:46:18.826499+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5327753,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":42,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.755496+0000","last_change":"2024-04-16T07:46:09.905947+0000","last_active":"2024-04-16T07:46:21.755496+0000","last_peered":"2024-04-16T07:46:21.755496+0000","last_clean":"2024-04-16T07:46:21.755496+0000","last_became_active":"2024-04-16T07:46:09.900754+0000","last_became_peered":"2024-04-16T07:46:09.900754+0000","last_unstale":"2024-04-16T07:46:21.755496+0000","last_undegraded":"2024-04-16T07:46:21.755496+0000","last_fullsized":"2024-04-16T07:46:21.755496+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"20'24","reported_seq":73,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.733418+0000","last_change":"2024-04-16T07:46:10.052676+0000","last_active":"2024-04-16T07:46:19.733418+0000","last_peered":"2024-04-16T07:46:19.733418+0000","last_clean":"2024-04-16T07:46:19.733418+0000","last_became_active":"2024-04-16T07:46:09.837421+0000","last_became_peered":"2024-04-16T07:46:09.837421+0000","last_unstale":"2024-04-16T07:46:19.733418+0000","last_undegraded":"2024-04-16T07:46:19.733418+0000","last_fullsized":"2024-04-16T07:46:19.733418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20991712,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":31,"num_write_kb":9471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":930388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'18","reported_seq":58,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.750437+0000","last_change":"2024-04-16T07:45:54.812310+0000","last_active":"2024-04-16T07:46:20.750437+0000","last_peered":"2024-04-16T07:46:20.750437+0000","last_clean":"2024-04-16T07:46:20.750437+0000","last_became_active":"2024-04-16T07:45:54.804593+0000","last_became_peered":"2024-04-16T07:45:54.804593+0000","last_unstale":"2024-04-16T07:46:20.750437+0000","last_undegraded":"2024-04-16T07:46:20.750437+0000","last_fullsized":"2024-04-16T07:46:20.750437+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14571504,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":23,"num_write_kb":6959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":44,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.743780+0000","last_change":"2024-04-16T07:46:09.886637+0000","last_active":"2024-04-16T07:46:19.743780+0000","last_peered":"2024-04-16T07:46:19.743780+0000","last_clean":"2024-04-16T07:46:19.743780+0000","last_became_active":"2024-04-16T07:46:09.882028+0000","last_became_peered":"2024-04-16T07:46:09.882028+0000","last_unstale":"2024-04-16T07:46:19.743780+0000","last_undegraded":"2024-04-16T07:46:19.743780+0000","last_fullsized":"2024-04-16T07:46:19.743780+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"20'20","reported_seq":68,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.734207+0000","last_change":"2024-04-16T07:46:09.984076+0000","last_active":"2024-04-16T07:46:19.734207+0000","last_peered":"2024-04-16T07:46:19.734207+0000","last_clean":"2024-04-16T07:46:19.734207+0000","last_became_active":"2024-04-16T07:46:09.828458+0000","last_became_peered":"2024-04-16T07:46:09.828458+0000","last_unstale":"2024-04-16T07:46:19.734207+0000","last_undegraded":"2024-04-16T07:46:19.734207+0000","last_fullsized":"2024-04-16T07:46:19.734207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17563117,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":26,"num_write_kb":7290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":41642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":45,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.836114+0000","last_change":"2024-04-16T07:46:09.857758+0000","last_active":"2024-04-16T07:46:18.836114+0000","last_peered":"2024-04-16T07:46:18.836114+0000","last_clean":"2024-04-16T07:46:18.836114+0000","last_became_active":"2024-04-16T07:46:09.850258+0000","last_became_peered":"2024-04-16T07:46:09.850258+0000","last_unstale":"2024-04-16T07:46:18.836114+0000","last_undegraded":"2024-04-16T07:46:18.836114+0000","last_fullsized":"2024-04-16T07:46:18.836114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'10","reported_seq":56,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.835308+0000","last_change":"2024-04-16T07:46:09.975356+0000","last_active":"2024-04-16T07:46:18.835308+0000","last_peered":"2024-04-16T07:46:18.835308+0000","last_clean":"2024-04-16T07:46:18.835308+0000","last_became_active":"2024-04-16T07:46:09.802912+0000","last_became_peered":"2024-04-16T07:46:09.802912+0000","last_unstale":"2024-04-16T07:46:18.835308+0000","last_undegraded":"2024-04-16T07:46:18.835308+0000","last_fullsized":"2024-04-16T07:46:18.835308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7428595,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":13,"num_write_kb":3079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":44,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.740283+0000","last_change":"2024-04-16T07:46:09.906633+0000","last_active":"2024-04-16T07:46:19.740283+0000","last_peered":"2024-04-16T07:46:19.740283+0000","last_clean":"2024-04-16T07:46:19.740283+0000","last_became_active":"2024-04-16T07:46:09.896255+0000","last_became_peered":"2024-04-16T07:46:09.896255+0000","last_unstale":"2024-04-16T07:46:19.740283+0000","last_undegraded":"2024-04-16T07:46:19.740283+0000","last_fullsized":"2024-04-16T07:46:19.740283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'19","reported_seq":61,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.729909+0000","last_change":"2024-04-16T07:45:54.819481+0000","last_active":"2024-04-16T07:46:19.729909+0000","last_peered":"2024-04-16T07:46:19.729909+0000","last_clean":"2024-04-16T07:46:19.729909+0000","last_became_active":"2024-04-16T07:45:54.813131+0000","last_became_peered":"2024-04-16T07:45:54.813131+0000","last_unstale":"2024-04-16T07:46:19.729909+0000","last_undegraded":"2024-04-16T07:46:19.729909+0000","last_fullsized":"2024-04-16T07:46:19.729909+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16184943,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":25,"num_write_kb":6850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":45,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.769122+0000","last_change":"2024-04-16T07:46:09.836581+0000","last_active":"2024-04-16T07:46:20.769122+0000","last_peered":"2024-04-16T07:46:20.769122+0000","last_clean":"2024-04-16T07:46:20.769122+0000","last_became_active":"2024-04-16T07:46:09.830316+0000","last_became_peered":"2024-04-16T07:46:09.830316+0000","last_unstale":"2024-04-16T07:46:20.769122+0000","last_undegraded":"2024-04-16T07:46:20.769122+0000","last_fullsized":"2024-04-16T07:46:20.769122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":213211653,"num_objects":79,"num_object_clones":0,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":155,"num_read_kb":76,"num_write":337,"num_write_kb":95765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2079924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":285,"ondisk_log_size":285,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705701,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":121520,"kb_used_data":94696,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94250320,"statfs":{"total":96636764160,"available":96512327680,"internally_reserved":0,"allocated":96968704,"data_stored":96608010,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":105888,"kb_used_data":79064,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94265952,"statfs":{"total":96636764160,"available":96528334848,"internally_reserved":0,"allocated":80961536,"data_stored":80730251,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705701,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99952,"kb_used_data":73128,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94271888,"statfs":{"total":96636764160,"available":96534413312,"internally_reserved":0,"allocated":74883072,"data_stored":74143430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378638,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99768,"kb_used_data":72816,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94272072,"statfs":{"total":96636764160,"available":96534601728,"internally_reserved":0,"allocated":74563584,"data_stored":74384382,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:46:25.047 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-16T07:46:25.047 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-04-16T07:46:25.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:25.156+0000 7f4839c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:25.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:25.165+0000 7f4839c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:25.366 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-16T07:46:25.375 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T07:46:25.375 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:46:25.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:25.484+0000 7fdf0f86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:25.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:25.492+0000 7fdf0f86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:25.679 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:46:25.679 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-16T07:46:25.692 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-04-16T07:46:25.627650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":213802021,"num_objects":81,"num_object_clones":0,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":258,"num_read_kb":167,"num_write":474,"num_write_kb":97659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2079924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":382,"ondisk_log_size":382,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":427128,"kb_used_data":319704,"kb_used_omap":0,"kb_used_meta":107392,"kb_avail":377060232,"statfs":{"total":386547056640,"available":386109677568,"internally_reserved":0,"allocated":327376896,"data_stored":325866073,"data_compressed":89727,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109969408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001482"},"pg_stats":[{"pgid":"3.f","version":"17'19","reported_seq":72,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.742216+0000","last_change":"2024-04-16T07:46:09.858371+0000","last_active":"2024-04-16T07:46:21.742216+0000","last_peered":"2024-04-16T07:46:21.742216+0000","last_clean":"2024-04-16T07:46:21.742216+0000","last_became_active":"2024-04-16T07:46:09.847369+0000","last_became_peered":"2024-04-16T07:46:09.847369+0000","last_unstale":"2024-04-16T07:46:21.742216+0000","last_undegraded":"2024-04-16T07:46:21.742216+0000","last_fullsized":"2024-04-16T07:46:21.742216+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16537890,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":25,"num_write_kb":7533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"17'27","reported_seq":71,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.729138+0000","last_change":"2024-04-16T07:45:54.897183+0000","last_active":"2024-04-16T07:46:19.729138+0000","last_peered":"2024-04-16T07:46:19.729138+0000","last_clean":"2024-04-16T07:46:19.729138+0000","last_became_active":"2024-04-16T07:45:54.891021+0000","last_became_peered":"2024-04-16T07:45:54.891021+0000","last_unstale":"2024-04-16T07:46:19.729138+0000","last_undegraded":"2024-04-16T07:46:19.729138+0000","last_fullsized":"2024-04-16T07:46:19.729138+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20353013,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":8,"num_write":35,"num_write_kb":8984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'20","reported_seq":67,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.768372+0000","last_change":"2024-04-16T07:46:10.109586+0000","last_active":"2024-04-16T07:46:20.768372+0000","last_peered":"2024-04-16T07:46:20.768372+0000","last_clean":"2024-04-16T07:46:20.768372+0000","last_became_active":"2024-04-16T07:46:09.820109+0000","last_became_peered":"2024-04-16T07:46:09.820109+0000","last_unstale":"2024-04-16T07:46:20.768372+0000","last_undegraded":"2024-04-16T07:46:20.768372+0000","last_fullsized":"2024-04-16T07:46:20.768372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16067294,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":26,"num_write_kb":7323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'32","reported_seq":89,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.754522+0000","last_change":"2024-04-16T07:46:09.896432+0000","last_active":"2024-04-16T07:46:21.754522+0000","last_peered":"2024-04-16T07:46:21.754522+0000","last_clean":"2024-04-16T07:46:21.754522+0000","last_became_active":"2024-04-16T07:46:09.891795+0000","last_became_peered":"2024-04-16T07:46:09.891795+0000","last_unstale":"2024-04-16T07:46:21.754522+0000","last_undegraded":"2024-04-16T07:46:21.754522+0000","last_fullsized":"2024-04-16T07:46:21.754522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25159749,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":9,"num_write":42,"num_write_kb":11798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'12","reported_seq":54,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.834431+0000","last_change":"2024-04-16T07:46:09.811202+0000","last_active":"2024-04-16T07:46:18.834431+0000","last_peered":"2024-04-16T07:46:18.834431+0000","last_clean":"2024-04-16T07:46:18.834431+0000","last_became_active":"2024-04-16T07:46:09.806014+0000","last_became_peered":"2024-04-16T07:46:09.806014+0000","last_unstale":"2024-04-16T07:46:18.834431+0000","last_undegraded":"2024-04-16T07:46:18.834431+0000","last_fullsized":"2024-04-16T07:46:18.834431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'5","reported_seq":54,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.753883+0000","last_change":"2024-04-16T07:46:09.873821+0000","last_active":"2024-04-16T07:46:21.753883+0000","last_peered":"2024-04-16T07:46:21.753883+0000","last_clean":"2024-04-16T07:46:21.753883+0000","last_became_active":"2024-04-16T07:46:09.869336+0000","last_became_peered":"2024-04-16T07:46:09.869336+0000","last_unstale":"2024-04-16T07:46:21.753883+0000","last_undegraded":"2024-04-16T07:46:21.753883+0000","last_fullsized":"2024-04-16T07:46:21.753883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4269570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":1976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'26","reported_seq":85,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.741032+0000","last_change":"2024-04-16T07:46:10.147324+0000","last_active":"2024-04-16T07:46:21.741032+0000","last_peered":"2024-04-16T07:46:21.741032+0000","last_clean":"2024-04-16T07:46:21.741032+0000","last_became_active":"2024-04-16T07:46:09.841067+0000","last_became_peered":"2024-04-16T07:46:09.841067+0000","last_unstale":"2024-04-16T07:46:21.741032+0000","last_undegraded":"2024-04-16T07:46:21.741032+0000","last_fullsized":"2024-04-16T07:46:21.741032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"17'23","reported_seq":65,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.759131+0000","last_change":"2024-04-16T07:45:54.797160+0000","last_active":"2024-04-16T07:46:20.759131+0000","last_peered":"2024-04-16T07:46:20.759131+0000","last_clean":"2024-04-16T07:46:20.759131+0000","last_became_active":"2024-04-16T07:45:54.786910+0000","last_became_peered":"2024-04-16T07:45:54.786910+0000","last_unstale":"2024-04-16T07:46:20.759131+0000","last_undegraded":"2024-04-16T07:46:20.759131+0000","last_fullsized":"2024-04-16T07:46:20.759131+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19294205,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":30,"num_write_kb":8200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":175,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.817843+0000","last_change":"2024-04-16T07:46:09.797708+0000","last_active":"2024-04-16T07:46:18.817843+0000","last_peered":"2024-04-16T07:46:18.817843+0000","last_clean":"2024-04-16T07:46:18.817843+0000","last_became_active":"2024-04-16T07:46:09.792221+0000","last_became_peered":"2024-04-16T07:46:09.792221+0000","last_unstale":"2024-04-16T07:46:18.817843+0000","last_undegraded":"2024-04-16T07:46:18.817843+0000","last_fullsized":"2024-04-16T07:46:18.817843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_clean_scrub_stamp":"2024-04-16T07:45:46.626705+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":45,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.772073+0000","last_change":"2024-04-16T07:46:09.796645+0000","last_active":"2024-04-16T07:46:20.772073+0000","last_peered":"2024-04-16T07:46:20.772073+0000","last_clean":"2024-04-16T07:46:20.772073+0000","last_became_active":"2024-04-16T07:46:09.790535+0000","last_became_peered":"2024-04-16T07:46:09.790535+0000","last_unstale":"2024-04-16T07:46:20.772073+0000","last_undegraded":"2024-04-16T07:46:20.772073+0000","last_fullsized":"2024-04-16T07:46:20.772073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"20'10","reported_seq":54,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.766709+0000","last_change":"2024-04-16T07:46:10.004761+0000","last_active":"2024-04-16T07:46:20.766709+0000","last_peered":"2024-04-16T07:46:20.766709+0000","last_clean":"2024-04-16T07:46:20.766709+0000","last_became_active":"2024-04-16T07:46:09.806452+0000","last_became_peered":"2024-04-16T07:46:09.806452+0000","last_unstale":"2024-04-16T07:46:20.766709+0000","last_undegraded":"2024-04-16T07:46:20.766709+0000","last_fullsized":"2024-04-16T07:46:20.766709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8184910,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":13,"num_write_kb":3711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":626546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":42,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.745900+0000","last_change":"2024-04-16T07:46:09.913777+0000","last_active":"2024-04-16T07:46:21.745900+0000","last_peered":"2024-04-16T07:46:21.745900+0000","last_clean":"2024-04-16T07:46:21.745900+0000","last_became_active":"2024-04-16T07:46:09.909374+0000","last_became_peered":"2024-04-16T07:46:09.909374+0000","last_unstale":"2024-04-16T07:46:21.745900+0000","last_undegraded":"2024-04-16T07:46:21.745900+0000","last_fullsized":"2024-04-16T07:46:21.745900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"20'14","reported_seq":61,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.829023+0000","last_change":"2024-04-16T07:46:10.087832+0000","last_active":"2024-04-16T07:46:18.829023+0000","last_peered":"2024-04-16T07:46:18.829023+0000","last_clean":"2024-04-16T07:46:18.829023+0000","last_became_active":"2024-04-16T07:46:09.828384+0000","last_became_peered":"2024-04-16T07:46:09.828384+0000","last_unstale":"2024-04-16T07:46:18.829023+0000","last_undegraded":"2024-04-16T07:46:18.829023+0000","last_fullsized":"2024-04-16T07:46:18.829023+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11462242,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":18,"num_write_kb":5591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":481348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":44,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.755443+0000","last_change":"2024-04-16T07:45:48.724954+0000","last_active":"2024-04-16T07:46:20.755443+0000","last_peered":"2024-04-16T07:46:20.755443+0000","last_clean":"2024-04-16T07:46:20.755443+0000","last_became_active":"2024-04-16T07:45:48.719985+0000","last_became_peered":"2024-04-16T07:45:48.719985+0000","last_unstale":"2024-04-16T07:46:20.755443+0000","last_undegraded":"2024-04-16T07:46:20.755443+0000","last_fullsized":"2024-04-16T07:46:20.755443+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'6","reported_seq":45,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.826499+0000","last_change":"2024-04-16T07:45:54.815123+0000","last_active":"2024-04-16T07:46:18.826499+0000","last_peered":"2024-04-16T07:46:18.826499+0000","last_clean":"2024-04-16T07:46:18.826499+0000","last_became_active":"2024-04-16T07:45:54.810589+0000","last_became_peered":"2024-04-16T07:45:54.810589+0000","last_unstale":"2024-04-16T07:46:18.826499+0000","last_undegraded":"2024-04-16T07:46:18.826499+0000","last_fullsized":"2024-04-16T07:46:18.826499+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5327753,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":42,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.755496+0000","last_change":"2024-04-16T07:46:09.905947+0000","last_active":"2024-04-16T07:46:21.755496+0000","last_peered":"2024-04-16T07:46:21.755496+0000","last_clean":"2024-04-16T07:46:21.755496+0000","last_became_active":"2024-04-16T07:46:09.900754+0000","last_became_peered":"2024-04-16T07:46:09.900754+0000","last_unstale":"2024-04-16T07:46:21.755496+0000","last_undegraded":"2024-04-16T07:46:21.755496+0000","last_fullsized":"2024-04-16T07:46:21.755496+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"20'24","reported_seq":73,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.733418+0000","last_change":"2024-04-16T07:46:10.052676+0000","last_active":"2024-04-16T07:46:19.733418+0000","last_peered":"2024-04-16T07:46:19.733418+0000","last_clean":"2024-04-16T07:46:19.733418+0000","last_became_active":"2024-04-16T07:46:09.837421+0000","last_became_peered":"2024-04-16T07:46:09.837421+0000","last_unstale":"2024-04-16T07:46:19.733418+0000","last_undegraded":"2024-04-16T07:46:19.733418+0000","last_fullsized":"2024-04-16T07:46:19.733418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20991712,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":31,"num_write_kb":9471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":930388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'18","reported_seq":58,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.750437+0000","last_change":"2024-04-16T07:45:54.812310+0000","last_active":"2024-04-16T07:46:20.750437+0000","last_peered":"2024-04-16T07:46:20.750437+0000","last_clean":"2024-04-16T07:46:20.750437+0000","last_became_active":"2024-04-16T07:45:54.804593+0000","last_became_peered":"2024-04-16T07:45:54.804593+0000","last_unstale":"2024-04-16T07:46:20.750437+0000","last_undegraded":"2024-04-16T07:46:20.750437+0000","last_fullsized":"2024-04-16T07:46:20.750437+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14571504,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":23,"num_write_kb":6959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":44,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.743780+0000","last_change":"2024-04-16T07:46:09.886637+0000","last_active":"2024-04-16T07:46:19.743780+0000","last_peered":"2024-04-16T07:46:19.743780+0000","last_clean":"2024-04-16T07:46:19.743780+0000","last_became_active":"2024-04-16T07:46:09.882028+0000","last_became_peered":"2024-04-16T07:46:09.882028+0000","last_unstale":"2024-04-16T07:46:19.743780+0000","last_undegraded":"2024-04-16T07:46:19.743780+0000","last_fullsized":"2024-04-16T07:46:19.743780+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"20'20","reported_seq":68,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.734207+0000","last_change":"2024-04-16T07:46:09.984076+0000","last_active":"2024-04-16T07:46:19.734207+0000","last_peered":"2024-04-16T07:46:19.734207+0000","last_clean":"2024-04-16T07:46:19.734207+0000","last_became_active":"2024-04-16T07:46:09.828458+0000","last_became_peered":"2024-04-16T07:46:09.828458+0000","last_unstale":"2024-04-16T07:46:19.734207+0000","last_undegraded":"2024-04-16T07:46:19.734207+0000","last_fullsized":"2024-04-16T07:46:19.734207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17563117,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":26,"num_write_kb":7290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":41642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":45,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.836114+0000","last_change":"2024-04-16T07:46:09.857758+0000","last_active":"2024-04-16T07:46:18.836114+0000","last_peered":"2024-04-16T07:46:18.836114+0000","last_clean":"2024-04-16T07:46:18.836114+0000","last_became_active":"2024-04-16T07:46:09.850258+0000","last_became_peered":"2024-04-16T07:46:09.850258+0000","last_unstale":"2024-04-16T07:46:18.836114+0000","last_undegraded":"2024-04-16T07:46:18.836114+0000","last_fullsized":"2024-04-16T07:46:18.836114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'10","reported_seq":56,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.835308+0000","last_change":"2024-04-16T07:46:09.975356+0000","last_active":"2024-04-16T07:46:18.835308+0000","last_peered":"2024-04-16T07:46:18.835308+0000","last_clean":"2024-04-16T07:46:18.835308+0000","last_became_active":"2024-04-16T07:46:09.802912+0000","last_became_peered":"2024-04-16T07:46:09.802912+0000","last_unstale":"2024-04-16T07:46:18.835308+0000","last_undegraded":"2024-04-16T07:46:18.835308+0000","last_fullsized":"2024-04-16T07:46:18.835308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7428595,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":13,"num_write_kb":3079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":44,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.740283+0000","last_change":"2024-04-16T07:46:09.906633+0000","last_active":"2024-04-16T07:46:19.740283+0000","last_peered":"2024-04-16T07:46:19.740283+0000","last_clean":"2024-04-16T07:46:19.740283+0000","last_became_active":"2024-04-16T07:46:09.896255+0000","last_became_peered":"2024-04-16T07:46:09.896255+0000","last_unstale":"2024-04-16T07:46:19.740283+0000","last_undegraded":"2024-04-16T07:46:19.740283+0000","last_fullsized":"2024-04-16T07:46:19.740283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'19","reported_seq":61,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.729909+0000","last_change":"2024-04-16T07:45:54.819481+0000","last_active":"2024-04-16T07:46:19.729909+0000","last_peered":"2024-04-16T07:46:19.729909+0000","last_clean":"2024-04-16T07:46:19.729909+0000","last_became_active":"2024-04-16T07:45:54.813131+0000","last_became_peered":"2024-04-16T07:45:54.813131+0000","last_unstale":"2024-04-16T07:46:19.729909+0000","last_undegraded":"2024-04-16T07:46:19.729909+0000","last_fullsized":"2024-04-16T07:46:19.729909+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16184943,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":25,"num_write_kb":6850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":45,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.769122+0000","last_change":"2024-04-16T07:46:09.836581+0000","last_active":"2024-04-16T07:46:20.769122+0000","last_peered":"2024-04-16T07:46:20.769122+0000","last_clean":"2024-04-16T07:46:20.769122+0000","last_became_active":"2024-04-16T07:46:09.830316+0000","last_became_peered":"2024-04-16T07:46:09.830316+0000","last_unstale":"2024-04-16T07:46:20.769122+0000","last_undegraded":"2024-04-16T07:46:20.769122+0000","last_fullsized":"2024-04-16T07:46:20.769122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":213211653,"num_objects":79,"num_object_clones":0,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":155,"num_read_kb":76,"num_write":337,"num_write_kb":95765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2079924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":285,"ondisk_log_size":285,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":121520,"kb_used_data":94696,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94250320,"statfs":{"total":96636764160,"available":96512327680,"internally_reserved":0,"allocated":96968704,"data_stored":96608010,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":105888,"kb_used_data":79064,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94265952,"statfs":{"total":96636764160,"available":96528334848,"internally_reserved":0,"allocated":80961536,"data_stored":80730251,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99952,"kb_used_data":73128,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94271888,"statfs":{"total":96636764160,"available":96534413312,"internally_reserved":0,"allocated":74883072,"data_stored":74143430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378638,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99768,"kb_used_data":72816,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94272072,"statfs":{"total":96636764160,"available":96534601728,"internally_reserved":0,"allocated":74563584,"data_stored":74384382,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:46:25.693 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:46:25.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:25.811+0000 7f7323ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:25.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:25.819+0000 7f7323ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:26.019 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:46:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-16T07:46:26.032 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-04-16T07:46:25.627650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":213802021,"num_objects":81,"num_object_clones":0,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":258,"num_read_kb":167,"num_write":474,"num_write_kb":97659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2079924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":382,"ondisk_log_size":382,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":427128,"kb_used_data":319704,"kb_used_omap":0,"kb_used_meta":107392,"kb_avail":377060232,"statfs":{"total":386547056640,"available":386109677568,"internally_reserved":0,"allocated":327376896,"data_stored":325866073,"data_compressed":89727,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109969408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001482"},"pg_stats":[{"pgid":"3.f","version":"17'19","reported_seq":72,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.742216+0000","last_change":"2024-04-16T07:46:09.858371+0000","last_active":"2024-04-16T07:46:21.742216+0000","last_peered":"2024-04-16T07:46:21.742216+0000","last_clean":"2024-04-16T07:46:21.742216+0000","last_became_active":"2024-04-16T07:46:09.847369+0000","last_became_peered":"2024-04-16T07:46:09.847369+0000","last_unstale":"2024-04-16T07:46:21.742216+0000","last_undegraded":"2024-04-16T07:46:21.742216+0000","last_fullsized":"2024-04-16T07:46:21.742216+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16537890,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":25,"num_write_kb":7533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"17'27","reported_seq":71,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.729138+0000","last_change":"2024-04-16T07:45:54.897183+0000","last_active":"2024-04-16T07:46:19.729138+0000","last_peered":"2024-04-16T07:46:19.729138+0000","last_clean":"2024-04-16T07:46:19.729138+0000","last_became_active":"2024-04-16T07:45:54.891021+0000","last_became_peered":"2024-04-16T07:45:54.891021+0000","last_unstale":"2024-04-16T07:46:19.729138+0000","last_undegraded":"2024-04-16T07:46:19.729138+0000","last_fullsized":"2024-04-16T07:46:19.729138+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20353013,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":8,"num_write":35,"num_write_kb":8984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'20","reported_seq":67,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.768372+0000","last_change":"2024-04-16T07:46:10.109586+0000","last_active":"2024-04-16T07:46:20.768372+0000","last_peered":"2024-04-16T07:46:20.768372+0000","last_clean":"2024-04-16T07:46:20.768372+0000","last_became_active":"2024-04-16T07:46:09.820109+0000","last_became_peered":"2024-04-16T07:46:09.820109+0000","last_unstale":"2024-04-16T07:46:20.768372+0000","last_undegraded":"2024-04-16T07:46:20.768372+0000","last_fullsized":"2024-04-16T07:46:20.768372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16067294,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":26,"num_write_kb":7323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'32","reported_seq":89,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.754522+0000","last_change":"2024-04-16T07:46:09.896432+0000","last_active":"2024-04-16T07:46:21.754522+0000","last_peered":"2024-04-16T07:46:21.754522+0000","last_clean":"2024-04-16T07:46:21.754522+0000","last_became_active":"2024-04-16T07:46:09.891795+0000","last_became_peered":"2024-04-16T07:46:09.891795+0000","last_unstale":"2024-04-16T07:46:21.754522+0000","last_undegraded":"2024-04-16T07:46:21.754522+0000","last_fullsized":"2024-04-16T07:46:21.754522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25159749,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":9,"num_write":42,"num_write_kb":11798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'12","reported_seq":54,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.834431+0000","last_change":"2024-04-16T07:46:09.811202+0000","last_active":"2024-04-16T07:46:18.834431+0000","last_peered":"2024-04-16T07:46:18.834431+0000","last_clean":"2024-04-16T07:46:18.834431+0000","last_became_active":"2024-04-16T07:46:09.806014+0000","last_became_peered":"2024-04-16T07:46:09.806014+0000","last_unstale":"2024-04-16T07:46:18.834431+0000","last_undegraded":"2024-04-16T07:46:18.834431+0000","last_fullsized":"2024-04-16T07:46:18.834431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'5","reported_seq":54,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.753883+0000","last_change":"2024-04-16T07:46:09.873821+0000","last_active":"2024-04-16T07:46:21.753883+0000","last_peered":"2024-04-16T07:46:21.753883+0000","last_clean":"2024-04-16T07:46:21.753883+0000","last_became_active":"2024-04-16T07:46:09.869336+0000","last_became_peered":"2024-04-16T07:46:09.869336+0000","last_unstale":"2024-04-16T07:46:21.753883+0000","last_undegraded":"2024-04-16T07:46:21.753883+0000","last_fullsized":"2024-04-16T07:46:21.753883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4269570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":1976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'26","reported_seq":85,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.741032+0000","last_change":"2024-04-16T07:46:10.147324+0000","last_active":"2024-04-16T07:46:21.741032+0000","last_peered":"2024-04-16T07:46:21.741032+0000","last_clean":"2024-04-16T07:46:21.741032+0000","last_became_active":"2024-04-16T07:46:09.841067+0000","last_became_peered":"2024-04-16T07:46:09.841067+0000","last_unstale":"2024-04-16T07:46:21.741032+0000","last_undegraded":"2024-04-16T07:46:21.741032+0000","last_fullsized":"2024-04-16T07:46:21.741032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"17'23","reported_seq":65,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.759131+0000","last_change":"2024-04-16T07:45:54.797160+0000","last_active":"2024-04-16T07:46:20.759131+0000","last_peered":"2024-04-16T07:46:20.759131+0000","last_clean":"2024-04-16T07:46:20.759131+0000","last_became_active":"2024-04-16T07:45:54.786910+0000","last_became_peered":"2024-04-16T07:45:54.786910+0000","last_unstale":"2024-04-16T07:46:20.759131+0000","last_undegraded":"2024-04-16T07:46:20.759131+0000","last_fullsized":"2024-04-16T07:46:20.759131+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19294205,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":30,"num_write_kb":8200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":175,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.817843+0000","last_change":"2024-04-16T07:46:09.797708+0000","last_active":"2024-04-16T07:46:18.817843+0000","last_peered":"2024-04-16T07:46:18.817843+0000","last_clean":"2024-04-16T07:46:18.817843+0000","last_became_active":"2024-04-16T07:46:09.792221+0000","last_became_peered":"2024-04-16T07:46:09.792221+0000","last_unstale":"2024-04-16T07:46:18.817843+0000","last_undegraded":"2024-04-16T07:46:18.817843+0000","last_fullsized":"2024-04-16T07:46:18.817843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_clean_scrub_stamp":"2024-04-16T07:45:46.626705+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":45,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.772073+0000","last_change":"2024-04-16T07:46:09.796645+0000","last_active":"2024-04-16T07:46:20.772073+0000","last_peered":"2024-04-16T07:46:20.772073+0000","last_clean":"2024-04-16T07:46:20.772073+0000","last_became_active":"2024-04-16T07:46:09.790535+0000","last_became_peered":"2024-04-16T07:46:09.790535+0000","last_unstale":"2024-04-16T07:46:20.772073+0000","last_undegraded":"2024-04-16T07:46:20.772073+0000","last_fullsized":"2024-04-16T07:46:20.772073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"20'10","reported_seq":54,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.766709+0000","last_change":"2024-04-16T07:46:10.004761+0000","last_active":"2024-04-16T07:46:20.766709+0000","last_peered":"2024-04-16T07:46:20.766709+0000","last_clean":"2024-04-16T07:46:20.766709+0000","last_became_active":"2024-04-16T07:46:09.806452+0000","last_became_peered":"2024-04-16T07:46:09.806452+0000","last_unstale":"2024-04-16T07:46:20.766709+0000","last_undegraded":"2024-04-16T07:46:20.766709+0000","last_fullsized":"2024-04-16T07:46:20.766709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8184910,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":13,"num_write_kb":3711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":626546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":42,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.745900+0000","last_change":"2024-04-16T07:46:09.913777+0000","last_active":"2024-04-16T07:46:21.745900+0000","last_peered":"2024-04-16T07:46:21.745900+0000","last_clean":"2024-04-16T07:46:21.745900+0000","last_became_active":"2024-04-16T07:46:09.909374+0000","last_became_peered":"2024-04-16T07:46:09.909374+0000","last_unstale":"2024-04-16T07:46:21.745900+0000","last_undegraded":"2024-04-16T07:46:21.745900+0000","last_fullsized":"2024-04-16T07:46:21.745900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"20'14","reported_seq":61,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.829023+0000","last_change":"2024-04-16T07:46:10.087832+0000","last_active":"2024-04-16T07:46:18.829023+0000","last_peered":"2024-04-16T07:46:18.829023+0000","last_clean":"2024-04-16T07:46:18.829023+0000","last_became_active":"2024-04-16T07:46:09.828384+0000","last_became_peered":"2024-04-16T07:46:09.828384+0000","last_unstale":"2024-04-16T07:46:18.829023+0000","last_undegraded":"2024-04-16T07:46:18.829023+0000","last_fullsized":"2024-04-16T07:46:18.829023+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11462242,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":18,"num_write_kb":5591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":481348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":44,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.755443+0000","last_change":"2024-04-16T07:45:48.724954+0000","last_active":"2024-04-16T07:46:20.755443+0000","last_peered":"2024-04-16T07:46:20.755443+0000","last_clean":"2024-04-16T07:46:20.755443+0000","last_became_active":"2024-04-16T07:45:48.719985+0000","last_became_peered":"2024-04-16T07:45:48.719985+0000","last_unstale":"2024-04-16T07:46:20.755443+0000","last_undegraded":"2024-04-16T07:46:20.755443+0000","last_fullsized":"2024-04-16T07:46:20.755443+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'6","reported_seq":45,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.826499+0000","last_change":"2024-04-16T07:45:54.815123+0000","last_active":"2024-04-16T07:46:18.826499+0000","last_peered":"2024-04-16T07:46:18.826499+0000","last_clean":"2024-04-16T07:46:18.826499+0000","last_became_active":"2024-04-16T07:45:54.810589+0000","last_became_peered":"2024-04-16T07:45:54.810589+0000","last_unstale":"2024-04-16T07:46:18.826499+0000","last_undegraded":"2024-04-16T07:46:18.826499+0000","last_fullsized":"2024-04-16T07:46:18.826499+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5327753,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":42,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.755496+0000","last_change":"2024-04-16T07:46:09.905947+0000","last_active":"2024-04-16T07:46:21.755496+0000","last_peered":"2024-04-16T07:46:21.755496+0000","last_clean":"2024-04-16T07:46:21.755496+0000","last_became_active":"2024-04-16T07:46:09.900754+0000","last_became_peered":"2024-04-16T07:46:09.900754+0000","last_unstale":"2024-04-16T07:46:21.755496+0000","last_undegraded":"2024-04-16T07:46:21.755496+0000","last_fullsized":"2024-04-16T07:46:21.755496+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"20'24","reported_seq":73,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.733418+0000","last_change":"2024-04-16T07:46:10.052676+0000","last_active":"2024-04-16T07:46:19.733418+0000","last_peered":"2024-04-16T07:46:19.733418+0000","last_clean":"2024-04-16T07:46:19.733418+0000","last_became_active":"2024-04-16T07:46:09.837421+0000","last_became_peered":"2024-04-16T07:46:09.837421+0000","last_unstale":"2024-04-16T07:46:19.733418+0000","last_undegraded":"2024-04-16T07:46:19.733418+0000","last_fullsized":"2024-04-16T07:46:19.733418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20991712,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":31,"num_write_kb":9471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":930388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'18","reported_seq":58,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.750437+0000","last_change":"2024-04-16T07:45:54.812310+0000","last_active":"2024-04-16T07:46:20.750437+0000","last_peered":"2024-04-16T07:46:20.750437+0000","last_clean":"2024-04-16T07:46:20.750437+0000","last_became_active":"2024-04-16T07:45:54.804593+0000","last_became_peered":"2024-04-16T07:45:54.804593+0000","last_unstale":"2024-04-16T07:46:20.750437+0000","last_undegraded":"2024-04-16T07:46:20.750437+0000","last_fullsized":"2024-04-16T07:46:20.750437+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14571504,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":23,"num_write_kb":6959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":44,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.743780+0000","last_change":"2024-04-16T07:46:09.886637+0000","last_active":"2024-04-16T07:46:19.743780+0000","last_peered":"2024-04-16T07:46:19.743780+0000","last_clean":"2024-04-16T07:46:19.743780+0000","last_became_active":"2024-04-16T07:46:09.882028+0000","last_became_peered":"2024-04-16T07:46:09.882028+0000","last_unstale":"2024-04-16T07:46:19.743780+0000","last_undegraded":"2024-04-16T07:46:19.743780+0000","last_fullsized":"2024-04-16T07:46:19.743780+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"20'20","reported_seq":68,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.734207+0000","last_change":"2024-04-16T07:46:09.984076+0000","last_active":"2024-04-16T07:46:19.734207+0000","last_peered":"2024-04-16T07:46:19.734207+0000","last_clean":"2024-04-16T07:46:19.734207+0000","last_became_active":"2024-04-16T07:46:09.828458+0000","last_became_peered":"2024-04-16T07:46:09.828458+0000","last_unstale":"2024-04-16T07:46:19.734207+0000","last_undegraded":"2024-04-16T07:46:19.734207+0000","last_fullsized":"2024-04-16T07:46:19.734207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17563117,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":26,"num_write_kb":7290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":41642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":45,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.836114+0000","last_change":"2024-04-16T07:46:09.857758+0000","last_active":"2024-04-16T07:46:18.836114+0000","last_peered":"2024-04-16T07:46:18.836114+0000","last_clean":"2024-04-16T07:46:18.836114+0000","last_became_active":"2024-04-16T07:46:09.850258+0000","last_became_peered":"2024-04-16T07:46:09.850258+0000","last_unstale":"2024-04-16T07:46:18.836114+0000","last_undegraded":"2024-04-16T07:46:18.836114+0000","last_fullsized":"2024-04-16T07:46:18.836114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'10","reported_seq":56,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.835308+0000","last_change":"2024-04-16T07:46:09.975356+0000","last_active":"2024-04-16T07:46:18.835308+0000","last_peered":"2024-04-16T07:46:18.835308+0000","last_clean":"2024-04-16T07:46:18.835308+0000","last_became_active":"2024-04-16T07:46:09.802912+0000","last_became_peered":"2024-04-16T07:46:09.802912+0000","last_unstale":"2024-04-16T07:46:18.835308+0000","last_undegraded":"2024-04-16T07:46:18.835308+0000","last_fullsized":"2024-04-16T07:46:18.835308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7428595,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":13,"num_write_kb":3079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":44,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.740283+0000","last_change":"2024-04-16T07:46:09.906633+0000","last_active":"2024-04-16T07:46:19.740283+0000","last_peered":"2024-04-16T07:46:19.740283+0000","last_clean":"2024-04-16T07:46:19.740283+0000","last_became_active":"2024-04-16T07:46:09.896255+0000","last_became_peered":"2024-04-16T07:46:09.896255+0000","last_unstale":"2024-04-16T07:46:19.740283+0000","last_undegraded":"2024-04-16T07:46:19.740283+0000","last_fullsized":"2024-04-16T07:46:19.740283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'19","reported_seq":61,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.729909+0000","last_change":"2024-04-16T07:45:54.819481+0000","last_active":"2024-04-16T07:46:19.729909+0000","last_peered":"2024-04-16T07:46:19.729909+0000","last_clean":"2024-04-16T07:46:19.729909+0000","last_became_active":"2024-04-16T07:45:54.813131+0000","last_became_peered":"2024-04-16T07:45:54.813131+0000","last_unstale":"2024-04-16T07:46:19.729909+0000","last_undegraded":"2024-04-16T07:46:19.729909+0000","last_fullsized":"2024-04-16T07:46:19.729909+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16184943,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":25,"num_write_kb":6850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":45,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.769122+0000","last_change":"2024-04-16T07:46:09.836581+0000","last_active":"2024-04-16T07:46:20.769122+0000","last_peered":"2024-04-16T07:46:20.769122+0000","last_clean":"2024-04-16T07:46:20.769122+0000","last_became_active":"2024-04-16T07:46:09.830316+0000","last_became_peered":"2024-04-16T07:46:09.830316+0000","last_unstale":"2024-04-16T07:46:20.769122+0000","last_undegraded":"2024-04-16T07:46:20.769122+0000","last_fullsized":"2024-04-16T07:46:20.769122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":213211653,"num_objects":79,"num_object_clones":0,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":155,"num_read_kb":76,"num_write":337,"num_write_kb":95765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2079924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":285,"ondisk_log_size":285,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":121520,"kb_used_data":94696,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94250320,"statfs":{"total":96636764160,"available":96512327680,"internally_reserved":0,"allocated":96968704,"data_stored":96608010,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":105888,"kb_used_data":79064,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94265952,"statfs":{"total":96636764160,"available":96528334848,"internally_reserved":0,"allocated":80961536,"data_stored":80730251,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99952,"kb_used_data":73128,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94271888,"statfs":{"total":96636764160,"available":96534413312,"internally_reserved":0,"allocated":74883072,"data_stored":74143430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378638,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99768,"kb_used_data":72816,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94272072,"statfs":{"total":96636764160,"available":96534601728,"internally_reserved":0,"allocated":74563584,"data_stored":74384382,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:46:26.033 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-16T07:46:26.034 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:46:26.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:26.143+0000 7f504455d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:26.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:26.150+0000 7f504455d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:26.334 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:46:26.335 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-16T07:46:26.347 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-04-16T07:46:25.627650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":213802021,"num_objects":81,"num_object_clones":0,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":258,"num_read_kb":167,"num_write":474,"num_write_kb":97659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2079924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":382,"ondisk_log_size":382,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":427128,"kb_used_data":319704,"kb_used_omap":0,"kb_used_meta":107392,"kb_avail":377060232,"statfs":{"total":386547056640,"available":386109677568,"internally_reserved":0,"allocated":327376896,"data_stored":325866073,"data_compressed":89727,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109969408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001482"},"pg_stats":[{"pgid":"3.f","version":"17'19","reported_seq":72,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.742216+0000","last_change":"2024-04-16T07:46:09.858371+0000","last_active":"2024-04-16T07:46:21.742216+0000","last_peered":"2024-04-16T07:46:21.742216+0000","last_clean":"2024-04-16T07:46:21.742216+0000","last_became_active":"2024-04-16T07:46:09.847369+0000","last_became_peered":"2024-04-16T07:46:09.847369+0000","last_unstale":"2024-04-16T07:46:21.742216+0000","last_undegraded":"2024-04-16T07:46:21.742216+0000","last_fullsized":"2024-04-16T07:46:21.742216+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16537890,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":25,"num_write_kb":7533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"17'27","reported_seq":71,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.729138+0000","last_change":"2024-04-16T07:45:54.897183+0000","last_active":"2024-04-16T07:46:19.729138+0000","last_peered":"2024-04-16T07:46:19.729138+0000","last_clean":"2024-04-16T07:46:19.729138+0000","last_became_active":"2024-04-16T07:45:54.891021+0000","last_became_peered":"2024-04-16T07:45:54.891021+0000","last_unstale":"2024-04-16T07:46:19.729138+0000","last_undegraded":"2024-04-16T07:46:19.729138+0000","last_fullsized":"2024-04-16T07:46:19.729138+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20353013,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":8,"num_write":35,"num_write_kb":8984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'20","reported_seq":67,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.768372+0000","last_change":"2024-04-16T07:46:10.109586+0000","last_active":"2024-04-16T07:46:20.768372+0000","last_peered":"2024-04-16T07:46:20.768372+0000","last_clean":"2024-04-16T07:46:20.768372+0000","last_became_active":"2024-04-16T07:46:09.820109+0000","last_became_peered":"2024-04-16T07:46:09.820109+0000","last_unstale":"2024-04-16T07:46:20.768372+0000","last_undegraded":"2024-04-16T07:46:20.768372+0000","last_fullsized":"2024-04-16T07:46:20.768372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16067294,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":26,"num_write_kb":7323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'32","reported_seq":89,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.754522+0000","last_change":"2024-04-16T07:46:09.896432+0000","last_active":"2024-04-16T07:46:21.754522+0000","last_peered":"2024-04-16T07:46:21.754522+0000","last_clean":"2024-04-16T07:46:21.754522+0000","last_became_active":"2024-04-16T07:46:09.891795+0000","last_became_peered":"2024-04-16T07:46:09.891795+0000","last_unstale":"2024-04-16T07:46:21.754522+0000","last_undegraded":"2024-04-16T07:46:21.754522+0000","last_fullsized":"2024-04-16T07:46:21.754522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25159749,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":9,"num_write":42,"num_write_kb":11798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'12","reported_seq":54,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.834431+0000","last_change":"2024-04-16T07:46:09.811202+0000","last_active":"2024-04-16T07:46:18.834431+0000","last_peered":"2024-04-16T07:46:18.834431+0000","last_clean":"2024-04-16T07:46:18.834431+0000","last_became_active":"2024-04-16T07:46:09.806014+0000","last_became_peered":"2024-04-16T07:46:09.806014+0000","last_unstale":"2024-04-16T07:46:18.834431+0000","last_undegraded":"2024-04-16T07:46:18.834431+0000","last_fullsized":"2024-04-16T07:46:18.834431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'5","reported_seq":54,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.753883+0000","last_change":"2024-04-16T07:46:09.873821+0000","last_active":"2024-04-16T07:46:21.753883+0000","last_peered":"2024-04-16T07:46:21.753883+0000","last_clean":"2024-04-16T07:46:21.753883+0000","last_became_active":"2024-04-16T07:46:09.869336+0000","last_became_peered":"2024-04-16T07:46:09.869336+0000","last_unstale":"2024-04-16T07:46:21.753883+0000","last_undegraded":"2024-04-16T07:46:21.753883+0000","last_fullsized":"2024-04-16T07:46:21.753883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4269570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":1976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'26","reported_seq":85,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.741032+0000","last_change":"2024-04-16T07:46:10.147324+0000","last_active":"2024-04-16T07:46:21.741032+0000","last_peered":"2024-04-16T07:46:21.741032+0000","last_clean":"2024-04-16T07:46:21.741032+0000","last_became_active":"2024-04-16T07:46:09.841067+0000","last_became_peered":"2024-04-16T07:46:09.841067+0000","last_unstale":"2024-04-16T07:46:21.741032+0000","last_undegraded":"2024-04-16T07:46:21.741032+0000","last_fullsized":"2024-04-16T07:46:21.741032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"17'23","reported_seq":65,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.759131+0000","last_change":"2024-04-16T07:45:54.797160+0000","last_active":"2024-04-16T07:46:20.759131+0000","last_peered":"2024-04-16T07:46:20.759131+0000","last_clean":"2024-04-16T07:46:20.759131+0000","last_became_active":"2024-04-16T07:45:54.786910+0000","last_became_peered":"2024-04-16T07:45:54.786910+0000","last_unstale":"2024-04-16T07:46:20.759131+0000","last_undegraded":"2024-04-16T07:46:20.759131+0000","last_fullsized":"2024-04-16T07:46:20.759131+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19294205,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":30,"num_write_kb":8200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":175,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.817843+0000","last_change":"2024-04-16T07:46:09.797708+0000","last_active":"2024-04-16T07:46:18.817843+0000","last_peered":"2024-04-16T07:46:18.817843+0000","last_clean":"2024-04-16T07:46:18.817843+0000","last_became_active":"2024-04-16T07:46:09.792221+0000","last_became_peered":"2024-04-16T07:46:09.792221+0000","last_unstale":"2024-04-16T07:46:18.817843+0000","last_undegraded":"2024-04-16T07:46:18.817843+0000","last_fullsized":"2024-04-16T07:46:18.817843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_clean_scrub_stamp":"2024-04-16T07:45:46.626705+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":45,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.772073+0000","last_change":"2024-04-16T07:46:09.796645+0000","last_active":"2024-04-16T07:46:20.772073+0000","last_peered":"2024-04-16T07:46:20.772073+0000","last_clean":"2024-04-16T07:46:20.772073+0000","last_became_active":"2024-04-16T07:46:09.790535+0000","last_became_peered":"2024-04-16T07:46:09.790535+0000","last_unstale":"2024-04-16T07:46:20.772073+0000","last_undegraded":"2024-04-16T07:46:20.772073+0000","last_fullsized":"2024-04-16T07:46:20.772073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"20'10","reported_seq":54,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.766709+0000","last_change":"2024-04-16T07:46:10.004761+0000","last_active":"2024-04-16T07:46:20.766709+0000","last_peered":"2024-04-16T07:46:20.766709+0000","last_clean":"2024-04-16T07:46:20.766709+0000","last_became_active":"2024-04-16T07:46:09.806452+0000","last_became_peered":"2024-04-16T07:46:09.806452+0000","last_unstale":"2024-04-16T07:46:20.766709+0000","last_undegraded":"2024-04-16T07:46:20.766709+0000","last_fullsized":"2024-04-16T07:46:20.766709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8184910,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":13,"num_write_kb":3711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":626546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":42,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.745900+0000","last_change":"2024-04-16T07:46:09.913777+0000","last_active":"2024-04-16T07:46:21.745900+0000","last_peered":"2024-04-16T07:46:21.745900+0000","last_clean":"2024-04-16T07:46:21.745900+0000","last_became_active":"2024-04-16T07:46:09.909374+0000","last_became_peered":"2024-04-16T07:46:09.909374+0000","last_unstale":"2024-04-16T07:46:21.745900+0000","last_undegraded":"2024-04-16T07:46:21.745900+0000","last_fullsized":"2024-04-16T07:46:21.745900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"20'14","reported_seq":61,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.829023+0000","last_change":"2024-04-16T07:46:10.087832+0000","last_active":"2024-04-16T07:46:18.829023+0000","last_peered":"2024-04-16T07:46:18.829023+0000","last_clean":"2024-04-16T07:46:18.829023+0000","last_became_active":"2024-04-16T07:46:09.828384+0000","last_became_peered":"2024-04-16T07:46:09.828384+0000","last_unstale":"2024-04-16T07:46:18.829023+0000","last_undegraded":"2024-04-16T07:46:18.829023+0000","last_fullsized":"2024-04-16T07:46:18.829023+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11462242,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":18,"num_write_kb":5591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":481348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":44,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.755443+0000","last_change":"2024-04-16T07:45:48.724954+0000","last_active":"2024-04-16T07:46:20.755443+0000","last_peered":"2024-04-16T07:46:20.755443+0000","last_clean":"2024-04-16T07:46:20.755443+0000","last_became_active":"2024-04-16T07:45:48.719985+0000","last_became_peered":"2024-04-16T07:45:48.719985+0000","last_unstale":"2024-04-16T07:46:20.755443+0000","last_undegraded":"2024-04-16T07:46:20.755443+0000","last_fullsized":"2024-04-16T07:46:20.755443+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'6","reported_seq":45,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.826499+0000","last_change":"2024-04-16T07:45:54.815123+0000","last_active":"2024-04-16T07:46:18.826499+0000","last_peered":"2024-04-16T07:46:18.826499+0000","last_clean":"2024-04-16T07:46:18.826499+0000","last_became_active":"2024-04-16T07:45:54.810589+0000","last_became_peered":"2024-04-16T07:45:54.810589+0000","last_unstale":"2024-04-16T07:46:18.826499+0000","last_undegraded":"2024-04-16T07:46:18.826499+0000","last_fullsized":"2024-04-16T07:46:18.826499+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5327753,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":42,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.755496+0000","last_change":"2024-04-16T07:46:09.905947+0000","last_active":"2024-04-16T07:46:21.755496+0000","last_peered":"2024-04-16T07:46:21.755496+0000","last_clean":"2024-04-16T07:46:21.755496+0000","last_became_active":"2024-04-16T07:46:09.900754+0000","last_became_peered":"2024-04-16T07:46:09.900754+0000","last_unstale":"2024-04-16T07:46:21.755496+0000","last_undegraded":"2024-04-16T07:46:21.755496+0000","last_fullsized":"2024-04-16T07:46:21.755496+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"20'24","reported_seq":73,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.733418+0000","last_change":"2024-04-16T07:46:10.052676+0000","last_active":"2024-04-16T07:46:19.733418+0000","last_peered":"2024-04-16T07:46:19.733418+0000","last_clean":"2024-04-16T07:46:19.733418+0000","last_became_active":"2024-04-16T07:46:09.837421+0000","last_became_peered":"2024-04-16T07:46:09.837421+0000","last_unstale":"2024-04-16T07:46:19.733418+0000","last_undegraded":"2024-04-16T07:46:19.733418+0000","last_fullsized":"2024-04-16T07:46:19.733418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20991712,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":31,"num_write_kb":9471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":930388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'18","reported_seq":58,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.750437+0000","last_change":"2024-04-16T07:45:54.812310+0000","last_active":"2024-04-16T07:46:20.750437+0000","last_peered":"2024-04-16T07:46:20.750437+0000","last_clean":"2024-04-16T07:46:20.750437+0000","last_became_active":"2024-04-16T07:45:54.804593+0000","last_became_peered":"2024-04-16T07:45:54.804593+0000","last_unstale":"2024-04-16T07:46:20.750437+0000","last_undegraded":"2024-04-16T07:46:20.750437+0000","last_fullsized":"2024-04-16T07:46:20.750437+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14571504,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":23,"num_write_kb":6959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":44,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.743780+0000","last_change":"2024-04-16T07:46:09.886637+0000","last_active":"2024-04-16T07:46:19.743780+0000","last_peered":"2024-04-16T07:46:19.743780+0000","last_clean":"2024-04-16T07:46:19.743780+0000","last_became_active":"2024-04-16T07:46:09.882028+0000","last_became_peered":"2024-04-16T07:46:09.882028+0000","last_unstale":"2024-04-16T07:46:19.743780+0000","last_undegraded":"2024-04-16T07:46:19.743780+0000","last_fullsized":"2024-04-16T07:46:19.743780+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"20'20","reported_seq":68,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.734207+0000","last_change":"2024-04-16T07:46:09.984076+0000","last_active":"2024-04-16T07:46:19.734207+0000","last_peered":"2024-04-16T07:46:19.734207+0000","last_clean":"2024-04-16T07:46:19.734207+0000","last_became_active":"2024-04-16T07:46:09.828458+0000","last_became_peered":"2024-04-16T07:46:09.828458+0000","last_unstale":"2024-04-16T07:46:19.734207+0000","last_undegraded":"2024-04-16T07:46:19.734207+0000","last_fullsized":"2024-04-16T07:46:19.734207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17563117,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":26,"num_write_kb":7290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":41642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":45,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.836114+0000","last_change":"2024-04-16T07:46:09.857758+0000","last_active":"2024-04-16T07:46:18.836114+0000","last_peered":"2024-04-16T07:46:18.836114+0000","last_clean":"2024-04-16T07:46:18.836114+0000","last_became_active":"2024-04-16T07:46:09.850258+0000","last_became_peered":"2024-04-16T07:46:09.850258+0000","last_unstale":"2024-04-16T07:46:18.836114+0000","last_undegraded":"2024-04-16T07:46:18.836114+0000","last_fullsized":"2024-04-16T07:46:18.836114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'10","reported_seq":56,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.835308+0000","last_change":"2024-04-16T07:46:09.975356+0000","last_active":"2024-04-16T07:46:18.835308+0000","last_peered":"2024-04-16T07:46:18.835308+0000","last_clean":"2024-04-16T07:46:18.835308+0000","last_became_active":"2024-04-16T07:46:09.802912+0000","last_became_peered":"2024-04-16T07:46:09.802912+0000","last_unstale":"2024-04-16T07:46:18.835308+0000","last_undegraded":"2024-04-16T07:46:18.835308+0000","last_fullsized":"2024-04-16T07:46:18.835308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7428595,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":13,"num_write_kb":3079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":44,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.740283+0000","last_change":"2024-04-16T07:46:09.906633+0000","last_active":"2024-04-16T07:46:19.740283+0000","last_peered":"2024-04-16T07:46:19.740283+0000","last_clean":"2024-04-16T07:46:19.740283+0000","last_became_active":"2024-04-16T07:46:09.896255+0000","last_became_peered":"2024-04-16T07:46:09.896255+0000","last_unstale":"2024-04-16T07:46:19.740283+0000","last_undegraded":"2024-04-16T07:46:19.740283+0000","last_fullsized":"2024-04-16T07:46:19.740283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'19","reported_seq":61,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.729909+0000","last_change":"2024-04-16T07:45:54.819481+0000","last_active":"2024-04-16T07:46:19.729909+0000","last_peered":"2024-04-16T07:46:19.729909+0000","last_clean":"2024-04-16T07:46:19.729909+0000","last_became_active":"2024-04-16T07:45:54.813131+0000","last_became_peered":"2024-04-16T07:45:54.813131+0000","last_unstale":"2024-04-16T07:46:19.729909+0000","last_undegraded":"2024-04-16T07:46:19.729909+0000","last_fullsized":"2024-04-16T07:46:19.729909+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16184943,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":25,"num_write_kb":6850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":45,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.769122+0000","last_change":"2024-04-16T07:46:09.836581+0000","last_active":"2024-04-16T07:46:20.769122+0000","last_peered":"2024-04-16T07:46:20.769122+0000","last_clean":"2024-04-16T07:46:20.769122+0000","last_became_active":"2024-04-16T07:46:09.830316+0000","last_became_peered":"2024-04-16T07:46:09.830316+0000","last_unstale":"2024-04-16T07:46:20.769122+0000","last_undegraded":"2024-04-16T07:46:20.769122+0000","last_fullsized":"2024-04-16T07:46:20.769122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":213211653,"num_objects":79,"num_object_clones":0,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":155,"num_read_kb":76,"num_write":337,"num_write_kb":95765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2079924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":285,"ondisk_log_size":285,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":121520,"kb_used_data":94696,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94250320,"statfs":{"total":96636764160,"available":96512327680,"internally_reserved":0,"allocated":96968704,"data_stored":96608010,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":105888,"kb_used_data":79064,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94265952,"statfs":{"total":96636764160,"available":96528334848,"internally_reserved":0,"allocated":80961536,"data_stored":80730251,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99952,"kb_used_data":73128,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94271888,"statfs":{"total":96636764160,"available":96534413312,"internally_reserved":0,"allocated":74883072,"data_stored":74143430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378638,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99768,"kb_used_data":72816,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94272072,"statfs":{"total":96636764160,"available":96534601728,"internally_reserved":0,"allocated":74563584,"data_stored":74384382,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:46:26.348 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:46:26.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:26.457+0000 7f79b5d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:26.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:26.464+0000 7f79b5d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:26.652 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:46:26.652 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":28,"fsid":"e0eb4f14-86a9-4014-9a92-cbfd9a7ee9c6","created":"2024-04-16T07:45:37.265201+0000","modified":"2024-04-16T07:46:21.724325+0000","last_up_change":"2024-04-16T07:46:08.680981+0000","last_in_change":"2024-04-16T07:45:39.445562+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:45:45.641324+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T07:45:47.599173+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-04-16T07:45:53.412511+0000","flags":262305,"flags_names":"hashpspool,nopgchange,write_fadvise_dontneed,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1,"score_stable":1,"optimal_score":1,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"816d6f55-bf1d-48a9-9552-fc285b77358e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":19,"up_thru":19,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":4131536773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6801","nonce":4131536773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":4131536773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6803","nonce":4131536773}]},"public_addr":"172.21.15.70:6800/4131536773","cluster_addr":"172.21.15.70:6801/4131536773","heartbeat_back_addr":"172.21.15.70:6808/4131536773","heartbeat_front_addr":"172.21.15.70:6803/4131536773","state":["exists","up"]},{"osd":1,"uuid":"f777184a-794a-4817-ba31-872e243adcd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":873227738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6805","nonce":873227738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":873227738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6807","nonce":873227738}]},"public_addr":"172.21.15.70:6804/873227738","cluster_addr":"172.21.15.70:6805/873227738","heartbeat_back_addr":"172.21.15.70:6806/873227738","heartbeat_front_addr":"172.21.15.70:6807/873227738","state":["exists","up"]},{"osd":2,"uuid":"340d1b05-a919-45a7-bc75-1cb38943c8e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":911620790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":911620790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":911620790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":911620790}]},"public_addr":"172.21.15.172:6804/911620790","cluster_addr":"172.21.15.172:6805/911620790","heartbeat_back_addr":"172.21.15.172:6810/911620790","heartbeat_front_addr":"172.21.15.172:6809/911620790","state":["exists","up"]},{"osd":3,"uuid":"659133e9-2b60-43c8-9034-f4690c837fe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1749870805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6807","nonce":1749870805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":1749870805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6803","nonce":1749870805}]},"public_addr":"172.21.15.172:6808/1749870805","cluster_addr":"172.21.15.172:6807/1749870805","heartbeat_back_addr":"172.21.15.172:6806/1749870805","heartbeat_front_addr":"172.21.15.172:6803/1749870805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2024-04-16T07:46:07.678267+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:46:27.666 INFO:tasks.ceph:Scrubbing osd.0 2024-04-16T07:46:27.667 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-16T07:46:27.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:27.784+0000 7fe9f9d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:27.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:27.796+0000 7fe9f9d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:27.809 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-16T07:46:27.809 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "" 2024-04-16T07:46:27.809 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-16T07:46:27.821 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-16T07:46:27.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:27.931+0000 7fd813d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:27.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:27.940+0000 7fd813d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:28.127 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 0 to deep-scrub 2024-04-16T07:46:28.148 INFO:tasks.ceph:Scrubbing osd.1 2024-04-16T07:46:28.148 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-16T07:46:28.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:28.318+0000 7fc37782d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:28.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:28.332+0000 7fc37782d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:28.350 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-16T07:46:28.350 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "" 2024-04-16T07:46:28.350 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-16T07:46:28.363 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-16T07:46:28.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:28.472+0000 7f1959406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:28.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:28.481+0000 7f1959406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:28.669 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 1 to deep-scrub 2024-04-16T07:46:28.695 INFO:tasks.ceph:Scrubbing osd.2 2024-04-16T07:46:28.695 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-16T07:46:28.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:28.843+0000 7f418bc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:28.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:28.858+0000 7f418bc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:28.878 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-16T07:46:28.878 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "" 2024-04-16T07:46:28.878 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-16T07:46:28.891 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-16T07:46:29.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:28.999+0000 7f2a212ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:29.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:29.008+0000 7f2a212ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:29.195 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 2 to deep-scrub 2024-04-16T07:46:29.208 INFO:tasks.ceph:Scrubbing osd.3 2024-04-16T07:46:29.209 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-16T07:46:29.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:29.348+0000 7f07302ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:29.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:29.360+0000 7f07302ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:29.430 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-16T07:46:29.430 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "" 2024-04-16T07:46:29.430 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-16T07:46:29.440 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-16T07:46:29.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:29.549+0000 7f429b060640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:29.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:29.558+0000 7f429b060640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:29.744 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 3 to deep-scrub 2024-04-16T07:46:29.757 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:46:29.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:29.885+0000 7fb33f8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:29.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:29.893+0000 7fb33f8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:30.078 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:46:30.078 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-16T07:46:30.090 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-04-16T07:46:29.628606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":213802021,"num_objects":81,"num_object_clones":0,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":258,"num_read_kb":167,"num_write":474,"num_write_kb":97659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2079924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":382,"ondisk_log_size":382,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":427128,"kb_used_data":319704,"kb_used_omap":0,"kb_used_meta":107392,"kb_avail":377060232,"statfs":{"total":386547056640,"available":386109677568,"internally_reserved":0,"allocated":327376896,"data_stored":325866073,"data_compressed":89727,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109969408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.907697"},"pg_stats":[{"pgid":"3.f","version":"17'19","reported_seq":72,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.742216+0000","last_change":"2024-04-16T07:46:09.858371+0000","last_active":"2024-04-16T07:46:21.742216+0000","last_peered":"2024-04-16T07:46:21.742216+0000","last_clean":"2024-04-16T07:46:21.742216+0000","last_became_active":"2024-04-16T07:46:09.847369+0000","last_became_peered":"2024-04-16T07:46:09.847369+0000","last_unstale":"2024-04-16T07:46:21.742216+0000","last_undegraded":"2024-04-16T07:46:21.742216+0000","last_fullsized":"2024-04-16T07:46:21.742216+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16537890,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":25,"num_write_kb":7533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"17'27","reported_seq":71,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.729138+0000","last_change":"2024-04-16T07:45:54.897183+0000","last_active":"2024-04-16T07:46:19.729138+0000","last_peered":"2024-04-16T07:46:19.729138+0000","last_clean":"2024-04-16T07:46:19.729138+0000","last_became_active":"2024-04-16T07:45:54.891021+0000","last_became_peered":"2024-04-16T07:45:54.891021+0000","last_unstale":"2024-04-16T07:46:19.729138+0000","last_undegraded":"2024-04-16T07:46:19.729138+0000","last_fullsized":"2024-04-16T07:46:19.729138+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20353013,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":8,"num_write":35,"num_write_kb":8984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'20","reported_seq":67,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.768372+0000","last_change":"2024-04-16T07:46:10.109586+0000","last_active":"2024-04-16T07:46:20.768372+0000","last_peered":"2024-04-16T07:46:20.768372+0000","last_clean":"2024-04-16T07:46:20.768372+0000","last_became_active":"2024-04-16T07:46:09.820109+0000","last_became_peered":"2024-04-16T07:46:09.820109+0000","last_unstale":"2024-04-16T07:46:20.768372+0000","last_undegraded":"2024-04-16T07:46:20.768372+0000","last_fullsized":"2024-04-16T07:46:20.768372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16067294,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":26,"num_write_kb":7323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'32","reported_seq":89,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.754522+0000","last_change":"2024-04-16T07:46:09.896432+0000","last_active":"2024-04-16T07:46:21.754522+0000","last_peered":"2024-04-16T07:46:21.754522+0000","last_clean":"2024-04-16T07:46:21.754522+0000","last_became_active":"2024-04-16T07:46:09.891795+0000","last_became_peered":"2024-04-16T07:46:09.891795+0000","last_unstale":"2024-04-16T07:46:21.754522+0000","last_undegraded":"2024-04-16T07:46:21.754522+0000","last_fullsized":"2024-04-16T07:46:21.754522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25159749,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":9,"num_write":42,"num_write_kb":11798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'12","reported_seq":54,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.834431+0000","last_change":"2024-04-16T07:46:09.811202+0000","last_active":"2024-04-16T07:46:18.834431+0000","last_peered":"2024-04-16T07:46:18.834431+0000","last_clean":"2024-04-16T07:46:18.834431+0000","last_became_active":"2024-04-16T07:46:09.806014+0000","last_became_peered":"2024-04-16T07:46:09.806014+0000","last_unstale":"2024-04-16T07:46:18.834431+0000","last_undegraded":"2024-04-16T07:46:18.834431+0000","last_fullsized":"2024-04-16T07:46:18.834431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'5","reported_seq":54,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.753883+0000","last_change":"2024-04-16T07:46:09.873821+0000","last_active":"2024-04-16T07:46:21.753883+0000","last_peered":"2024-04-16T07:46:21.753883+0000","last_clean":"2024-04-16T07:46:21.753883+0000","last_became_active":"2024-04-16T07:46:09.869336+0000","last_became_peered":"2024-04-16T07:46:09.869336+0000","last_unstale":"2024-04-16T07:46:21.753883+0000","last_undegraded":"2024-04-16T07:46:21.753883+0000","last_fullsized":"2024-04-16T07:46:21.753883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4269570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":1976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'26","reported_seq":85,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.741032+0000","last_change":"2024-04-16T07:46:10.147324+0000","last_active":"2024-04-16T07:46:21.741032+0000","last_peered":"2024-04-16T07:46:21.741032+0000","last_clean":"2024-04-16T07:46:21.741032+0000","last_became_active":"2024-04-16T07:46:09.841067+0000","last_became_peered":"2024-04-16T07:46:09.841067+0000","last_unstale":"2024-04-16T07:46:21.741032+0000","last_undegraded":"2024-04-16T07:46:21.741032+0000","last_fullsized":"2024-04-16T07:46:21.741032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"17'23","reported_seq":65,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.759131+0000","last_change":"2024-04-16T07:45:54.797160+0000","last_active":"2024-04-16T07:46:20.759131+0000","last_peered":"2024-04-16T07:46:20.759131+0000","last_clean":"2024-04-16T07:46:20.759131+0000","last_became_active":"2024-04-16T07:45:54.786910+0000","last_became_peered":"2024-04-16T07:45:54.786910+0000","last_unstale":"2024-04-16T07:46:20.759131+0000","last_undegraded":"2024-04-16T07:46:20.759131+0000","last_fullsized":"2024-04-16T07:46:20.759131+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19294205,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":30,"num_write_kb":8200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":175,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.817843+0000","last_change":"2024-04-16T07:46:09.797708+0000","last_active":"2024-04-16T07:46:18.817843+0000","last_peered":"2024-04-16T07:46:18.817843+0000","last_clean":"2024-04-16T07:46:18.817843+0000","last_became_active":"2024-04-16T07:46:09.792221+0000","last_became_peered":"2024-04-16T07:46:09.792221+0000","last_unstale":"2024-04-16T07:46:18.817843+0000","last_undegraded":"2024-04-16T07:46:18.817843+0000","last_fullsized":"2024-04-16T07:46:18.817843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:46.626705+0000","last_clean_scrub_stamp":"2024-04-16T07:45:46.626705+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":45,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.772073+0000","last_change":"2024-04-16T07:46:09.796645+0000","last_active":"2024-04-16T07:46:20.772073+0000","last_peered":"2024-04-16T07:46:20.772073+0000","last_clean":"2024-04-16T07:46:20.772073+0000","last_became_active":"2024-04-16T07:46:09.790535+0000","last_became_peered":"2024-04-16T07:46:09.790535+0000","last_unstale":"2024-04-16T07:46:20.772073+0000","last_undegraded":"2024-04-16T07:46:20.772073+0000","last_fullsized":"2024-04-16T07:46:20.772073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"20'10","reported_seq":54,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.766709+0000","last_change":"2024-04-16T07:46:10.004761+0000","last_active":"2024-04-16T07:46:20.766709+0000","last_peered":"2024-04-16T07:46:20.766709+0000","last_clean":"2024-04-16T07:46:20.766709+0000","last_became_active":"2024-04-16T07:46:09.806452+0000","last_became_peered":"2024-04-16T07:46:09.806452+0000","last_unstale":"2024-04-16T07:46:20.766709+0000","last_undegraded":"2024-04-16T07:46:20.766709+0000","last_fullsized":"2024-04-16T07:46:20.766709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8184910,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":13,"num_write_kb":3711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":626546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":42,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.745900+0000","last_change":"2024-04-16T07:46:09.913777+0000","last_active":"2024-04-16T07:46:21.745900+0000","last_peered":"2024-04-16T07:46:21.745900+0000","last_clean":"2024-04-16T07:46:21.745900+0000","last_became_active":"2024-04-16T07:46:09.909374+0000","last_became_peered":"2024-04-16T07:46:09.909374+0000","last_unstale":"2024-04-16T07:46:21.745900+0000","last_undegraded":"2024-04-16T07:46:21.745900+0000","last_fullsized":"2024-04-16T07:46:21.745900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"20'14","reported_seq":61,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.829023+0000","last_change":"2024-04-16T07:46:10.087832+0000","last_active":"2024-04-16T07:46:18.829023+0000","last_peered":"2024-04-16T07:46:18.829023+0000","last_clean":"2024-04-16T07:46:18.829023+0000","last_became_active":"2024-04-16T07:46:09.828384+0000","last_became_peered":"2024-04-16T07:46:09.828384+0000","last_unstale":"2024-04-16T07:46:18.829023+0000","last_undegraded":"2024-04-16T07:46:18.829023+0000","last_fullsized":"2024-04-16T07:46:18.829023+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11462242,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":18,"num_write_kb":5591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":481348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":44,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.755443+0000","last_change":"2024-04-16T07:45:48.724954+0000","last_active":"2024-04-16T07:46:20.755443+0000","last_peered":"2024-04-16T07:46:20.755443+0000","last_clean":"2024-04-16T07:46:20.755443+0000","last_became_active":"2024-04-16T07:45:48.719985+0000","last_became_peered":"2024-04-16T07:45:48.719985+0000","last_unstale":"2024-04-16T07:46:20.755443+0000","last_undegraded":"2024-04-16T07:46:20.755443+0000","last_fullsized":"2024-04-16T07:46:20.755443+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'6","reported_seq":45,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.826499+0000","last_change":"2024-04-16T07:45:54.815123+0000","last_active":"2024-04-16T07:46:18.826499+0000","last_peered":"2024-04-16T07:46:18.826499+0000","last_clean":"2024-04-16T07:46:18.826499+0000","last_became_active":"2024-04-16T07:45:54.810589+0000","last_became_peered":"2024-04-16T07:45:54.810589+0000","last_unstale":"2024-04-16T07:46:18.826499+0000","last_undegraded":"2024-04-16T07:46:18.826499+0000","last_fullsized":"2024-04-16T07:46:18.826499+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5327753,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":42,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:21.755496+0000","last_change":"2024-04-16T07:46:09.905947+0000","last_active":"2024-04-16T07:46:21.755496+0000","last_peered":"2024-04-16T07:46:21.755496+0000","last_clean":"2024-04-16T07:46:21.755496+0000","last_became_active":"2024-04-16T07:46:09.900754+0000","last_became_peered":"2024-04-16T07:46:09.900754+0000","last_unstale":"2024-04-16T07:46:21.755496+0000","last_undegraded":"2024-04-16T07:46:21.755496+0000","last_fullsized":"2024-04-16T07:46:21.755496+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"20'24","reported_seq":73,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.733418+0000","last_change":"2024-04-16T07:46:10.052676+0000","last_active":"2024-04-16T07:46:19.733418+0000","last_peered":"2024-04-16T07:46:19.733418+0000","last_clean":"2024-04-16T07:46:19.733418+0000","last_became_active":"2024-04-16T07:46:09.837421+0000","last_became_peered":"2024-04-16T07:46:09.837421+0000","last_unstale":"2024-04-16T07:46:19.733418+0000","last_undegraded":"2024-04-16T07:46:19.733418+0000","last_fullsized":"2024-04-16T07:46:19.733418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20991712,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":31,"num_write_kb":9471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":930388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'18","reported_seq":58,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.750437+0000","last_change":"2024-04-16T07:45:54.812310+0000","last_active":"2024-04-16T07:46:20.750437+0000","last_peered":"2024-04-16T07:46:20.750437+0000","last_clean":"2024-04-16T07:46:20.750437+0000","last_became_active":"2024-04-16T07:45:54.804593+0000","last_became_peered":"2024-04-16T07:45:54.804593+0000","last_unstale":"2024-04-16T07:46:20.750437+0000","last_undegraded":"2024-04-16T07:46:20.750437+0000","last_fullsized":"2024-04-16T07:46:20.750437+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14571504,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":23,"num_write_kb":6959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":44,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.743780+0000","last_change":"2024-04-16T07:46:09.886637+0000","last_active":"2024-04-16T07:46:19.743780+0000","last_peered":"2024-04-16T07:46:19.743780+0000","last_clean":"2024-04-16T07:46:19.743780+0000","last_became_active":"2024-04-16T07:46:09.882028+0000","last_became_peered":"2024-04-16T07:46:09.882028+0000","last_unstale":"2024-04-16T07:46:19.743780+0000","last_undegraded":"2024-04-16T07:46:19.743780+0000","last_fullsized":"2024-04-16T07:46:19.743780+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"20'20","reported_seq":68,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.734207+0000","last_change":"2024-04-16T07:46:09.984076+0000","last_active":"2024-04-16T07:46:19.734207+0000","last_peered":"2024-04-16T07:46:19.734207+0000","last_clean":"2024-04-16T07:46:19.734207+0000","last_became_active":"2024-04-16T07:46:09.828458+0000","last_became_peered":"2024-04-16T07:46:09.828458+0000","last_unstale":"2024-04-16T07:46:19.734207+0000","last_undegraded":"2024-04-16T07:46:19.734207+0000","last_fullsized":"2024-04-16T07:46:19.734207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17563117,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":26,"num_write_kb":7290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":41642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":45,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.836114+0000","last_change":"2024-04-16T07:46:09.857758+0000","last_active":"2024-04-16T07:46:18.836114+0000","last_peered":"2024-04-16T07:46:18.836114+0000","last_clean":"2024-04-16T07:46:18.836114+0000","last_became_active":"2024-04-16T07:46:09.850258+0000","last_became_peered":"2024-04-16T07:46:09.850258+0000","last_unstale":"2024-04-16T07:46:18.836114+0000","last_undegraded":"2024-04-16T07:46:18.836114+0000","last_fullsized":"2024-04-16T07:46:18.836114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'10","reported_seq":56,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-16T07:46:18.835308+0000","last_change":"2024-04-16T07:46:09.975356+0000","last_active":"2024-04-16T07:46:18.835308+0000","last_peered":"2024-04-16T07:46:18.835308+0000","last_clean":"2024-04-16T07:46:18.835308+0000","last_became_active":"2024-04-16T07:46:09.802912+0000","last_became_peered":"2024-04-16T07:46:09.802912+0000","last_unstale":"2024-04-16T07:46:18.835308+0000","last_undegraded":"2024-04-16T07:46:18.835308+0000","last_fullsized":"2024-04-16T07:46:18.835308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7428595,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":13,"num_write_kb":3079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":44,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.740283+0000","last_change":"2024-04-16T07:46:09.906633+0000","last_active":"2024-04-16T07:46:19.740283+0000","last_peered":"2024-04-16T07:46:19.740283+0000","last_clean":"2024-04-16T07:46:19.740283+0000","last_became_active":"2024-04-16T07:46:09.896255+0000","last_became_peered":"2024-04-16T07:46:09.896255+0000","last_unstale":"2024-04-16T07:46:19.740283+0000","last_undegraded":"2024-04-16T07:46:19.740283+0000","last_fullsized":"2024-04-16T07:46:19.740283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'19","reported_seq":61,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-16T07:46:19.729909+0000","last_change":"2024-04-16T07:45:54.819481+0000","last_active":"2024-04-16T07:46:19.729909+0000","last_peered":"2024-04-16T07:46:19.729909+0000","last_clean":"2024-04-16T07:46:19.729909+0000","last_became_active":"2024-04-16T07:45:54.813131+0000","last_became_peered":"2024-04-16T07:45:54.813131+0000","last_unstale":"2024-04-16T07:46:19.729909+0000","last_undegraded":"2024-04-16T07:46:19.729909+0000","last_fullsized":"2024-04-16T07:46:19.729909+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:53.655068+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16184943,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":25,"num_write_kb":6850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":45,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-16T07:46:20.769122+0000","last_change":"2024-04-16T07:46:09.836581+0000","last_active":"2024-04-16T07:46:20.769122+0000","last_peered":"2024-04-16T07:46:20.769122+0000","last_clean":"2024-04-16T07:46:20.769122+0000","last_became_active":"2024-04-16T07:46:09.830316+0000","last_became_peered":"2024-04-16T07:46:09.830316+0000","last_unstale":"2024-04-16T07:46:20.769122+0000","last_undegraded":"2024-04-16T07:46:20.769122+0000","last_fullsized":"2024-04-16T07:46:20.769122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:47.630254+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":213211653,"num_objects":79,"num_object_clones":0,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":155,"num_read_kb":76,"num_write":337,"num_write_kb":95765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2079924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":285,"ondisk_log_size":285,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":121520,"kb_used_data":94696,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94250320,"statfs":{"total":96636764160,"available":96512327680,"internally_reserved":0,"allocated":96968704,"data_stored":96608010,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":105888,"kb_used_data":79064,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94265952,"statfs":{"total":96636764160,"available":96528334848,"internally_reserved":0,"allocated":80961536,"data_stored":80730251,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99952,"kb_used_data":73128,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94271888,"statfs":{"total":96636764160,"available":96534413312,"internally_reserved":0,"allocated":74883072,"data_stored":74143430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378643,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99768,"kb_used_data":72816,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94272072,"statfs":{"total":96636764160,"available":96534601728,"internally_reserved":0,"allocated":74563584,"data_stored":74384382,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:46:30.094 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-16T07:45:53.655068+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=53, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.094 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-16T07:45:53.655068+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=53, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.094 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-04-16T07:45:53.655068+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=53, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.094 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-16T07:45:53.655068+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=53, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.094 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-04-16T07:45:53.655068+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=53, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.095 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-04-16T07:45:53.655068+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=53, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.095 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-04-16T07:45:53.655068+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=53, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.095 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-16T07:45:53.655068+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=53, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.095 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-16T07:45:46.626705+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.095 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-16T07:45:47.630254+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=47, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.095 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-16T07:45:53.655068+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=53, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.095 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-16T07:45:47.630254+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=47, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.095 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-16T07:45:53.655068+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=53, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.095 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-16T07:45:47.630254+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=47, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.095 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-16T07:45:53.655068+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=53, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.095 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-16T07:45:47.630254+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=47, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.096 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-16T07:45:53.655068+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=53, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.096 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-04-16T07:45:53.655068+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=53, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.096 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-16T07:45:47.630254+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=47, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.096 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-04-16T07:45:53.655068+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=53, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.096 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-16T07:45:47.630254+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=47, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.096 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-04-16T07:45:53.655068+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=53, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.096 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-16T07:45:47.630254+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=47, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.096 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-04-16T07:45:53.655068+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=53, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.096 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-16T07:45:47.630254+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=45, tm_sec=47, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=7, tm_min=46, tm_sec=26, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T07:46:30.096 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-16T07:46:50.098 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:46:50.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:50.214+0000 7f46e4864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:50.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:50.222+0000 7f46e4864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:50.408 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-16T07:46:50.408 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-16T07:46:50.420 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-04-16T07:46:49.633917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":213802021,"num_objects":81,"num_object_clones":0,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":258,"num_read_kb":167,"num_write":474,"num_write_kb":97659,"num_scrub_errors":0,"num_shallow_scrub_errors":2,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2079924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":382,"ondisk_log_size":382,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":427176,"kb_used_data":319752,"kb_used_omap":0,"kb_used_meta":107392,"kb_avail":377060184,"statfs":{"total":386547056640,"available":386109628416,"internally_reserved":0,"allocated":327426048,"data_stored":325885639,"data_compressed":89727,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109969408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003127"},"pg_stats":[{"pgid":"3.f","version":"17'19","reported_seq":74,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:28.133134+0000","last_change":"2024-04-16T07:46:28.362587+0000","last_active":"2024-04-16T07:46:28.133134+0000","last_peered":"2024-04-16T07:46:28.133134+0000","last_clean":"2024-04-16T07:46:28.133134+0000","last_became_active":"2024-04-16T07:46:09.847369+0000","last_became_peered":"2024-04-16T07:46:09.847369+0000","last_unstale":"2024-04-16T07:46:28.133134+0000","last_undegraded":"2024-04-16T07:46:28.133134+0000","last_fullsized":"2024-04-16T07:46:28.133134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'19","last_scrub_stamp":"2024-04-16T07:46:28.362363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:28.362363+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16537890,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":25,"num_write_kb":7533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"17'27","reported_seq":77,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:29.202886+0000","last_change":"2024-04-16T07:46:29.468191+0000","last_active":"2024-04-16T07:46:29.202886+0000","last_peered":"2024-04-16T07:46:29.202886+0000","last_clean":"2024-04-16T07:46:29.202886+0000","last_became_active":"2024-04-16T07:45:54.891021+0000","last_became_peered":"2024-04-16T07:45:54.891021+0000","last_unstale":"2024-04-16T07:46:29.202886+0000","last_undegraded":"2024-04-16T07:46:29.202886+0000","last_fullsized":"2024-04-16T07:46:29.202886+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'27","last_scrub_stamp":"2024-04-16T07:46:29.467915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:29.467915+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20353013,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":8,"num_write":35,"num_write_kb":8984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'20","reported_seq":72,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:28.682254+0000","last_change":"2024-04-16T07:46:28.899219+0000","last_active":"2024-04-16T07:46:28.682254+0000","last_peered":"2024-04-16T07:46:28.682254+0000","last_clean":"2024-04-16T07:46:28.682254+0000","last_became_active":"2024-04-16T07:46:09.820109+0000","last_became_peered":"2024-04-16T07:46:09.820109+0000","last_unstale":"2024-04-16T07:46:28.682254+0000","last_undegraded":"2024-04-16T07:46:28.682254+0000","last_fullsized":"2024-04-16T07:46:28.682254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'20","last_scrub_stamp":"2024-04-16T07:46:28.898975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:28.898975+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16067294,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":26,"num_write_kb":7323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'32","reported_seq":91,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:28.135361+0000","last_change":"2024-04-16T07:46:28.314422+0000","last_active":"2024-04-16T07:46:28.135361+0000","last_peered":"2024-04-16T07:46:28.135361+0000","last_clean":"2024-04-16T07:46:28.135361+0000","last_became_active":"2024-04-16T07:46:09.891795+0000","last_became_peered":"2024-04-16T07:46:09.891795+0000","last_unstale":"2024-04-16T07:46:28.135361+0000","last_undegraded":"2024-04-16T07:46:28.135361+0000","last_fullsized":"2024-04-16T07:46:28.135361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'32","last_scrub_stamp":"2024-04-16T07:46:28.314159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:28.314159+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25159749,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":9,"num_write":42,"num_write_kb":11798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'12","reported_seq":61,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:29.750751+0000","last_change":"2024-04-16T07:46:29.834879+0000","last_active":"2024-04-16T07:46:29.750751+0000","last_peered":"2024-04-16T07:46:29.750751+0000","last_clean":"2024-04-16T07:46:29.750751+0000","last_became_active":"2024-04-16T07:46:09.806014+0000","last_became_peered":"2024-04-16T07:46:09.806014+0000","last_unstale":"2024-04-16T07:46:29.750751+0000","last_undegraded":"2024-04-16T07:46:29.750751+0000","last_fullsized":"2024-04-16T07:46:29.750751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'12","last_scrub_stamp":"2024-04-16T07:46:29.834639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:29.834639+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'5","reported_seq":56,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:28.134262+0000","last_change":"2024-04-16T07:46:28.194648+0000","last_active":"2024-04-16T07:46:28.134262+0000","last_peered":"2024-04-16T07:46:28.134262+0000","last_clean":"2024-04-16T07:46:28.134262+0000","last_became_active":"2024-04-16T07:46:09.869336+0000","last_became_peered":"2024-04-16T07:46:09.869336+0000","last_unstale":"2024-04-16T07:46:28.134262+0000","last_undegraded":"2024-04-16T07:46:28.134262+0000","last_fullsized":"2024-04-16T07:46:28.134262+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'5","last_scrub_stamp":"2024-04-16T07:46:28.194355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:28.194355+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4269570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":1976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'26","reported_seq":87,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:28.131485+0000","last_change":"2024-04-16T07:46:28.385686+0000","last_active":"2024-04-16T07:46:28.131485+0000","last_peered":"2024-04-16T07:46:28.131485+0000","last_clean":"2024-04-16T07:46:28.131485+0000","last_became_active":"2024-04-16T07:46:09.841067+0000","last_became_peered":"2024-04-16T07:46:09.841067+0000","last_unstale":"2024-04-16T07:46:28.131485+0000","last_undegraded":"2024-04-16T07:46:28.131485+0000","last_fullsized":"2024-04-16T07:46:28.131485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'26","last_scrub_stamp":"2024-04-16T07:46:28.385466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:28.385466+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":2,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"17'23","reported_seq":70,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:28.678721+0000","last_change":"2024-04-16T07:46:28.895949+0000","last_active":"2024-04-16T07:46:28.678721+0000","last_peered":"2024-04-16T07:46:28.678721+0000","last_clean":"2024-04-16T07:46:28.678721+0000","last_became_active":"2024-04-16T07:45:54.786910+0000","last_became_peered":"2024-04-16T07:45:54.786910+0000","last_unstale":"2024-04-16T07:46:28.678721+0000","last_undegraded":"2024-04-16T07:46:28.678721+0000","last_fullsized":"2024-04-16T07:46:28.678721+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'23","last_scrub_stamp":"2024-04-16T07:46:28.895723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:28.895723+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19294205,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":30,"num_write_kb":8200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":182,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:29.747885+0000","last_change":"2024-04-16T07:46:29.822379+0000","last_active":"2024-04-16T07:46:29.747885+0000","last_peered":"2024-04-16T07:46:29.747885+0000","last_clean":"2024-04-16T07:46:29.747885+0000","last_became_active":"2024-04-16T07:46:09.792221+0000","last_became_peered":"2024-04-16T07:46:09.792221+0000","last_unstale":"2024-04-16T07:46:29.747885+0000","last_undegraded":"2024-04-16T07:46:29.747885+0000","last_fullsized":"2024-04-16T07:46:29.747885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'97","last_scrub_stamp":"2024-04-16T07:46:29.822128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:29.822128+0000","last_clean_scrub_stamp":"2024-04-16T07:45:46.626705+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":50,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:28.675802+0000","last_change":"2024-04-16T07:46:28.741211+0000","last_active":"2024-04-16T07:46:28.675802+0000","last_peered":"2024-04-16T07:46:28.675802+0000","last_clean":"2024-04-16T07:46:28.675802+0000","last_became_active":"2024-04-16T07:46:09.790535+0000","last_became_peered":"2024-04-16T07:46:09.790535+0000","last_unstale":"2024-04-16T07:46:28.675802+0000","last_undegraded":"2024-04-16T07:46:28.675802+0000","last_fullsized":"2024-04-16T07:46:28.675802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:46:28.740868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:28.740868+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"20'10","reported_seq":59,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:28.679019+0000","last_change":"2024-04-16T07:46:28.849559+0000","last_active":"2024-04-16T07:46:28.679019+0000","last_peered":"2024-04-16T07:46:28.679019+0000","last_clean":"2024-04-16T07:46:28.679019+0000","last_became_active":"2024-04-16T07:46:09.806452+0000","last_became_peered":"2024-04-16T07:46:09.806452+0000","last_unstale":"2024-04-16T07:46:28.679019+0000","last_undegraded":"2024-04-16T07:46:28.679019+0000","last_fullsized":"2024-04-16T07:46:28.679019+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'10","last_scrub_stamp":"2024-04-16T07:46:28.849169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:28.849169+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8184910,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":13,"num_write_kb":3711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":626546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":44,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:28.131932+0000","last_change":"2024-04-16T07:46:28.180059+0000","last_active":"2024-04-16T07:46:28.131932+0000","last_peered":"2024-04-16T07:46:28.131932+0000","last_clean":"2024-04-16T07:46:28.131932+0000","last_became_active":"2024-04-16T07:46:09.909374+0000","last_became_peered":"2024-04-16T07:46:09.909374+0000","last_unstale":"2024-04-16T07:46:28.131932+0000","last_undegraded":"2024-04-16T07:46:28.131932+0000","last_fullsized":"2024-04-16T07:46:28.131932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:46:28.179784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:28.179784+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"20'14","reported_seq":68,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:29.750443+0000","last_change":"2024-04-16T07:46:29.866306+0000","last_active":"2024-04-16T07:46:29.750443+0000","last_peered":"2024-04-16T07:46:29.750443+0000","last_clean":"2024-04-16T07:46:29.750443+0000","last_became_active":"2024-04-16T07:46:09.828384+0000","last_became_peered":"2024-04-16T07:46:09.828384+0000","last_unstale":"2024-04-16T07:46:29.750443+0000","last_undegraded":"2024-04-16T07:46:29.750443+0000","last_fullsized":"2024-04-16T07:46:29.750443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'14","last_scrub_stamp":"2024-04-16T07:46:29.866071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:29.866071+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11462242,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":18,"num_write_kb":5591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":481348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:28.673860+0000","last_change":"2024-04-16T07:46:28.712459+0000","last_active":"2024-04-16T07:46:28.673860+0000","last_peered":"2024-04-16T07:46:28.673860+0000","last_clean":"2024-04-16T07:46:28.673860+0000","last_became_active":"2024-04-16T07:45:48.719985+0000","last_became_peered":"2024-04-16T07:45:48.719985+0000","last_unstale":"2024-04-16T07:46:28.673860+0000","last_undegraded":"2024-04-16T07:46:28.673860+0000","last_fullsized":"2024-04-16T07:46:28.673860+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:46:28.712144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:28.712144+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'6","reported_seq":52,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:29.749976+0000","last_change":"2024-04-16T07:46:29.816613+0000","last_active":"2024-04-16T07:46:29.749976+0000","last_peered":"2024-04-16T07:46:29.749976+0000","last_clean":"2024-04-16T07:46:29.749976+0000","last_became_active":"2024-04-16T07:45:54.810589+0000","last_became_peered":"2024-04-16T07:45:54.810589+0000","last_unstale":"2024-04-16T07:46:29.749976+0000","last_undegraded":"2024-04-16T07:46:29.749976+0000","last_fullsized":"2024-04-16T07:46:29.749976+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'6","last_scrub_stamp":"2024-04-16T07:46:29.816379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:29.816379+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5327753,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":44,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:28.132916+0000","last_change":"2024-04-16T07:46:28.190211+0000","last_active":"2024-04-16T07:46:28.132916+0000","last_peered":"2024-04-16T07:46:28.132916+0000","last_clean":"2024-04-16T07:46:28.132916+0000","last_became_active":"2024-04-16T07:46:09.900754+0000","last_became_peered":"2024-04-16T07:46:09.900754+0000","last_unstale":"2024-04-16T07:46:28.132916+0000","last_undegraded":"2024-04-16T07:46:28.132916+0000","last_fullsized":"2024-04-16T07:46:28.132916+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:46:28.189872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:28.189872+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"20'24","reported_seq":79,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:29.199994+0000","last_change":"2024-04-16T07:46:29.452504+0000","last_active":"2024-04-16T07:46:29.199994+0000","last_peered":"2024-04-16T07:46:29.199994+0000","last_clean":"2024-04-16T07:46:29.199994+0000","last_became_active":"2024-04-16T07:46:09.837421+0000","last_became_peered":"2024-04-16T07:46:09.837421+0000","last_unstale":"2024-04-16T07:46:29.199994+0000","last_undegraded":"2024-04-16T07:46:29.199994+0000","last_fullsized":"2024-04-16T07:46:29.199994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'24","last_scrub_stamp":"2024-04-16T07:46:29.452265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:29.452265+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20991712,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":31,"num_write_kb":9471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":930388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'18","reported_seq":63,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:28.676924+0000","last_change":"2024-04-16T07:46:28.891124+0000","last_active":"2024-04-16T07:46:28.676924+0000","last_peered":"2024-04-16T07:46:28.676924+0000","last_clean":"2024-04-16T07:46:28.676924+0000","last_became_active":"2024-04-16T07:45:54.804593+0000","last_became_peered":"2024-04-16T07:45:54.804593+0000","last_unstale":"2024-04-16T07:46:28.676924+0000","last_undegraded":"2024-04-16T07:46:28.676924+0000","last_fullsized":"2024-04-16T07:46:28.676924+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'18","last_scrub_stamp":"2024-04-16T07:46:28.890884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:28.890884+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14571504,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":23,"num_write_kb":6959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":50,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:29.199816+0000","last_change":"2024-04-16T07:46:29.250328+0000","last_active":"2024-04-16T07:46:29.199816+0000","last_peered":"2024-04-16T07:46:29.199816+0000","last_clean":"2024-04-16T07:46:29.199816+0000","last_became_active":"2024-04-16T07:46:09.882028+0000","last_became_peered":"2024-04-16T07:46:09.882028+0000","last_unstale":"2024-04-16T07:46:29.199816+0000","last_undegraded":"2024-04-16T07:46:29.199816+0000","last_fullsized":"2024-04-16T07:46:29.199816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:46:29.250128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:29.250128+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"20'20","reported_seq":74,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:29.200947+0000","last_change":"2024-04-16T07:46:29.447645+0000","last_active":"2024-04-16T07:46:29.200947+0000","last_peered":"2024-04-16T07:46:29.200947+0000","last_clean":"2024-04-16T07:46:29.200947+0000","last_became_active":"2024-04-16T07:46:09.828458+0000","last_became_peered":"2024-04-16T07:46:09.828458+0000","last_unstale":"2024-04-16T07:46:29.200947+0000","last_undegraded":"2024-04-16T07:46:29.200947+0000","last_fullsized":"2024-04-16T07:46:29.200947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'20","last_scrub_stamp":"2024-04-16T07:46:29.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:29.447385+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17563117,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":26,"num_write_kb":7290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":41642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":52,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:29.748001+0000","last_change":"2024-04-16T07:46:29.781908+0000","last_active":"2024-04-16T07:46:29.748001+0000","last_peered":"2024-04-16T07:46:29.748001+0000","last_clean":"2024-04-16T07:46:29.748001+0000","last_became_active":"2024-04-16T07:46:09.850258+0000","last_became_peered":"2024-04-16T07:46:09.850258+0000","last_unstale":"2024-04-16T07:46:29.748001+0000","last_undegraded":"2024-04-16T07:46:29.748001+0000","last_fullsized":"2024-04-16T07:46:29.748001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:46:29.781601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:29.781601+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'10","reported_seq":63,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:29.749844+0000","last_change":"2024-04-16T07:46:29.844304+0000","last_active":"2024-04-16T07:46:29.749844+0000","last_peered":"2024-04-16T07:46:29.749844+0000","last_clean":"2024-04-16T07:46:29.749844+0000","last_became_active":"2024-04-16T07:46:09.802912+0000","last_became_peered":"2024-04-16T07:46:09.802912+0000","last_unstale":"2024-04-16T07:46:29.749844+0000","last_undegraded":"2024-04-16T07:46:29.749844+0000","last_fullsized":"2024-04-16T07:46:29.749844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'10","last_scrub_stamp":"2024-04-16T07:46:29.844052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:29.844052+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7428595,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":13,"num_write_kb":3079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":50,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:29.199814+0000","last_change":"2024-04-16T07:46:29.243309+0000","last_active":"2024-04-16T07:46:29.199814+0000","last_peered":"2024-04-16T07:46:29.199814+0000","last_clean":"2024-04-16T07:46:29.199814+0000","last_became_active":"2024-04-16T07:46:09.896255+0000","last_became_peered":"2024-04-16T07:46:09.896255+0000","last_unstale":"2024-04-16T07:46:29.199814+0000","last_undegraded":"2024-04-16T07:46:29.199814+0000","last_fullsized":"2024-04-16T07:46:29.199814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:46:29.242979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:29.242979+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'19","reported_seq":67,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:29.202179+0000","last_change":"2024-04-16T07:46:29.424804+0000","last_active":"2024-04-16T07:46:29.202179+0000","last_peered":"2024-04-16T07:46:29.202179+0000","last_clean":"2024-04-16T07:46:29.202179+0000","last_became_active":"2024-04-16T07:45:54.813131+0000","last_became_peered":"2024-04-16T07:45:54.813131+0000","last_unstale":"2024-04-16T07:46:29.202179+0000","last_undegraded":"2024-04-16T07:46:29.202179+0000","last_fullsized":"2024-04-16T07:46:29.202179+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'19","last_scrub_stamp":"2024-04-16T07:46:29.424542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:29.424542+0000","last_clean_scrub_stamp":"2024-04-16T07:45:53.655068+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16184943,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":25,"num_write_kb":6850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":50,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-16T07:46:28.678289+0000","last_change":"2024-04-16T07:46:28.731611+0000","last_active":"2024-04-16T07:46:28.678289+0000","last_peered":"2024-04-16T07:46:28.678289+0000","last_clean":"2024-04-16T07:46:28.678289+0000","last_became_active":"2024-04-16T07:46:09.830316+0000","last_became_peered":"2024-04-16T07:46:09.830316+0000","last_unstale":"2024-04-16T07:46:28.678289+0000","last_undegraded":"2024-04-16T07:46:28.678289+0000","last_fullsized":"2024-04-16T07:46:28.678289+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:46:28.731423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:46:28.731423+0000","last_clean_scrub_stamp":"2024-04-16T07:45:47.630254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":213211653,"num_objects":79,"num_object_clones":0,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":155,"num_read_kb":76,"num_write":337,"num_write_kb":95765,"num_scrub_errors":0,"num_shallow_scrub_errors":2,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2079924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":285,"ondisk_log_size":285,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":121544,"kb_used_data":94720,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94250296,"statfs":{"total":96636764160,"available":96512303104,"internally_reserved":0,"allocated":96993280,"data_stored":96617793,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":105904,"kb_used_data":79080,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94265936,"statfs":{"total":96636764160,"available":96528318464,"internally_reserved":0,"allocated":80977920,"data_stored":80736773,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99960,"kb_used_data":73136,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94271880,"statfs":{"total":96636764160,"available":96534405120,"internally_reserved":0,"allocated":74891264,"data_stored":74146691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378663,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99768,"kb_used_data":72816,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94272072,"statfs":{"total":96636764160,"available":96534601728,"internally_reserved":0,"allocated":74563584,"data_stored":74384382,"data_compressed":29909,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:46:50.423 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-16T07:46:50.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:50.529+0000 7f1dca39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:50.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-16T07:46:50.541+0000 7f1dca39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:50.748 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/rados.py", line 308, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/rados.py", line 289, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' 2024-04-16T07:46:50.749 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-16T07:46:50.749 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-16T07:46:50.749 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-16T07:46:50.749 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:46:50.829 INFO:tasks.ceph.osd.0:Stopped 2024-04-16T07:46:50.829 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-16T07:46:50.829 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:46:50.905 INFO:tasks.ceph.osd.1:Stopped 2024-04-16T07:46:50.905 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-16T07:46:50.905 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:46:50.973 INFO:tasks.ceph.osd.2:Stopped 2024-04-16T07:46:50.974 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-16T07:46:50.974 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:46:51.053 INFO:tasks.ceph.osd.3:Stopped 2024-04-16T07:46:51.054 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-16T07:46:51.054 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-16T07:46:51.054 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:46:51.080 INFO:tasks.ceph.mgr.x:Stopped 2024-04-16T07:46:51.080 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-16T07:46:51.080 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-16T07:46:51.080 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:46:51.091 INFO:tasks.ceph.mon.a:Stopped 2024-04-16T07:46:51.091 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-16T07:46:51.091 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-16T07:46:51.165 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-16T07:46:51.165 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-16T07:46:51.312 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-16T07:46:51.312 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-16T07:46:51.384 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi172.front.sepia.ceph.com 2024-04-16T07:46:51.384 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-16T07:46:51.481 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi172.front.sepia.ceph.com 2024-04-16T07:46:51.481 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-16T07:46:51.554 INFO:tasks.ceph:Archiving mon data... 2024-04-16T07:46:51.555 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657976/data/mon.a.tgz 2024-04-16T07:46:51.555 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-04-16T07:46:51.571 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.6z1LcVafuD 2024-04-16T07:46:51.571 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.6z1LcVafuD 2024-04-16T07:46:51.682 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.6z1LcVafuD 2024-04-16T07:46:51.769 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.6z1LcVafuD is 167KB 2024-04-16T07:46:51.820 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.6z1LcVafuD 2024-04-16T07:46:51.835 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-16T07:46:51.835 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T07:46:51.876 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T07:46:51.956 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-16T07:46:51.972 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-16T07:46:51.973 INFO:tasks.ceph:Archiving crash dumps... 2024-04-16T07:46:51.974 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657976/remote/smithi070/crash 2024-04-16T07:46:51.975 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-16T07:46:52.024 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657976/remote/smithi172/crash 2024-04-16T07:46:52.025 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-16T07:46:52.099 INFO:tasks.ceph:Compressing logs... 2024-04-16T07:46:52.099 DEBUG:teuthology.orchestra.run.smithi070:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T07:46:52.102 DEBUG:teuthology.orchestra.run.smithi172:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T07:46:52.136 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29583.log 2024-04-16T07:46:52.136 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29503.log 2024-04-16T07:46:52.136 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29543.log 2024-04-16T07:46:52.136 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29583.log: 49.6%/var/log/ceph/ceph-client.admin.29503.log: -- replaced with /var/log/ceph/ceph-client.admin.29583.log.gz 2024-04-16T07:46:52.136 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-16T07:46:52.136 INFO:teuthology.orchestra.run.smithi172.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29503.log.gz 2024-04-16T07:46:52.136 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29623.log 2024-04-16T07:46:52.136 INFO:teuthology.orchestra.run.smithi172.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29543.log.gz 2024-04-16T07:46:52.136 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-16T07:46:52.136 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29623.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-16T07:46:52.136 INFO:teuthology.orchestra.run.smithi172.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29623.log.gz 2024-04-16T07:46:52.151 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-osd.2.log: 91.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-16T07:46:52.159 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30642.log 2024-04-16T07:46:52.159 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30365.log 2024-04-16T07:46:52.159 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.30642.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-16T07:46:52.159 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30642.log.gz 2024-04-16T07:46:52.159 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.30365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33202.log 2024-04-16T07:46:52.159 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30365.log.gz 2024-04-16T07:46:52.159 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31555.log 2024-04-16T07:46:52.159 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32109.log 2024-04-16T07:46:52.159 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33202.log.gz 2024-04-16T07:46:52.159 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.31555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31003.log 2024-04-16T07:46:52.159 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31555.log.gz 2024-04-16T07:46:52.159 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30643.log 2024-04-16T07:46:52.159 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32109.log.gz 2024-04-16T07:46:52.159 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.31003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32067.log 2024-04-16T07:46:52.160 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31003.log.gz 2024-04-16T07:46:52.160 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.30643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30810.log 2024-04-16T07:46:52.160 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30643.log.gz 2024-04-16T07:46:52.160 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32908.log 2024-04-16T07:46:52.160 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32067.log.gz 2024-04-16T07:46:52.160 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.30810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32277.log 2024-04-16T07:46:52.160 INFO:teuthology.orchestra.run.smithi070.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30810.log.gz 2024-04-16T07:46:52.160 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30407.log 2024-04-16T07:46:52.160 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32908.log.gz 2024-04-16T07:46:52.160 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31899.log 2024-04-16T07:46:52.160 INFO:teuthology.orchestra.run.smithi070.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32277.log.gz 2024-04-16T07:46:52.160 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.30407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30919.log 2024-04-16T07:46:52.160 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30407.log.gz 2024-04-16T07:46:52.160 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.31899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33497.log 2024-04-16T07:46:52.160 INFO:teuthology.orchestra.run.smithi070.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31899.log.gz 2024-04-16T07:46:52.161 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.30919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32377.log 2024-04-16T07:46:52.161 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30919.log.gz 2024-04-16T07:46:52.161 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31087.log 2024-04-16T07:46:52.161 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33497.log.gz 2024-04-16T07:46:52.161 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32950.log 2024-04-16T07:46:52.161 INFO:teuthology.orchestra.run.smithi070.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32377.log.gz 2024-04-16T07:46:52.161 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.31087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31856.log 2024-04-16T07:46:52.161 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31087.log.gz 2024-04-16T07:46:52.161 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32235.log 2024-04-16T07:46:52.161 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32950.log.gz 2024-04-16T07:46:52.161 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.31856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32527.log 2024-04-16T07:46:52.161 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31856.log.gz 2024-04-16T07:46:52.161 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32235.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-16T07:46:52.161 INFO:teuthology.orchestra.run.smithi070.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32235.log.gz 2024-04-16T07:46:52.161 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32736.log 2024-04-16T07:46:52.161 INFO:teuthology.orchestra.run.smithi070.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32527.log.gz 2024-04-16T07:46:52.162 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31470.log 2024-04-16T07:46:52.162 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33244.log 2024-04-16T07:46:52.162 INFO:teuthology.orchestra.run.smithi070.stderr: 87.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-16T07:46:52.162 INFO:teuthology.orchestra.run.smithi070.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32736.log.gz 2024-04-16T07:46:52.162 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.31470.log: gzip 50.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.31470.log.gz --verbose 2024-04-16T07:46:52.162 INFO:teuthology.orchestra.run.smithi070.stderr: -- /var/log/ceph/ceph-client.admin.29821.log 2024-04-16T07:46:52.162 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32824.log 2024-04-16T07:46:52.162 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33244.log.gz 2024-04-16T07:46:52.162 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.29821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30499.log 2024-04-16T07:46:52.162 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29821.log.gz 2024-04-16T07:46:52.162 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30309.log 2024-04-16T07:46:52.162 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32824.log.gz 2024-04-16T07:46:52.162 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.30499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33328.log 2024-04-16T07:46:52.162 INFO:teuthology.orchestra.run.smithi070.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30499.log.gz 2024-04-16T07:46:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.30309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33160.log 2024-04-16T07:46:52.163 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30309.log.gz 2024-04-16T07:46:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33328.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-16T07:46:52.163 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33328.log.gz 2024-04-16T07:46:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31171.log 2024-04-16T07:46:52.163 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33160.log.gz 2024-04-16T07:46:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose --/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.30640.log 2024-04-16T07:46:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.31171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31807.log 2024-04-16T07:46:52.163 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31171.log.gz 2024-04-16T07:46:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.30640.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30640.log.gz 2024-04-16T07:46:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33076.log 2024-04-16T07:46:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.31807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32025.log 2024-04-16T07:46:52.163 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31807.log.gz 2024-04-16T07:46:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.31500.log 2024-04-16T07:46:52.163 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33076.log.gz 2024-04-16T07:46:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33118.log 2024-04-16T07:46:52.164 INFO:teuthology.orchestra.run.smithi070.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32025.log.gz 2024-04-16T07:46:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.0.31500.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph-client.admin.32193.log -- replaced with /var/log/ceph/ceph-client.0.31500.log.gz 2024-04-16T07:46:52.164 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-16T07:46:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32656.log 2024-04-16T07:46:52.164 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33118.log.gz 2024-04-16T07:46:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32193.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32193.log.gzgzip 2024-04-16T07:46:52.164 INFO:teuthology.orchestra.run.smithi070.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30811.log 2024-04-16T07:46:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31233.log 2024-04-16T07:46:52.164 INFO:teuthology.orchestra.run.smithi070.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32656.log.gz 2024-04-16T07:46:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.30811.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30811.log.gz 2024-04-16T07:46:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31983.log 2024-04-16T07:46:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31332.log 2024-04-16T07:46:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.31233.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31233.log.gz 2024-04-16T07:46:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.31983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30449.log 2024-04-16T07:46:52.164 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31983.log.gz 2024-04-16T07:46:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.31332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32488.log 2024-04-16T07:46:52.165 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31332.log.gz 2024-04-16T07:46:52.165 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.30449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32528.log 2024-04-16T07:46:52.165 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30449.log.gz 2024-04-16T07:46:52.165 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30086.log 2024-04-16T07:46:52.165 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32488.log.gz 2024-04-16T07:46:52.165 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33454.log 2024-04-16T07:46:52.165 INFO:teuthology.orchestra.run.smithi070.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32528.log.gz 2024-04-16T07:46:52.165 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.30086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30058.log 2024-04-16T07:46:52.165 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30086.log.gz 2024-04-16T07:46:52.165 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33454.log: 49.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.33454.log.gz -5 2024-04-16T07:46:52.165 INFO:teuthology.orchestra.run.smithi070.stderr: --verbose -- /var/log/ceph/ceph-client.admin.32151.log 2024-04-16T07:46:52.165 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.30058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32676.log 2024-04-16T07:46:52.165 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30058.log.gz 2024-04-16T07:46:52.165 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32151.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32151.log.gz 2024-04-16T07:46:52.165 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31941.log 2024-04-16T07:46:52.165 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33412.log 2024-04-16T07:46:52.165 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32676.log.gz 2024-04-16T07:46:52.166 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.31941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33538.log 2024-04-16T07:46:52.166 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31941.log.gz 2024-04-16T07:46:52.166 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31605.log 2024-04-16T07:46:52.166 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33412.log.gz 2024-04-16T07:46:52.166 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31330.log 2024-04-16T07:46:52.166 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33538.log.gz 2024-04-16T07:46:52.166 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.31605.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-16T07:46:52.166 INFO:teuthology.orchestra.run.smithi070.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31605.log.gz 2024-04-16T07:46:52.166 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.31330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30809.log 2024-04-16T07:46:52.166 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31330.log.gz 2024-04-16T07:46:52.166 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31129.log 2024-04-16T07:46:52.166 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.30809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32698.log 2024-04-16T07:46:52.166 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30809.log.gz 2024-04-16T07:46:52.166 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.31129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33370.log 2024-04-16T07:46:52.166 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31129.log.gz 2024-04-16T07:46:52.166 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30641.log 2024-04-16T07:46:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32698.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32698.log.gz 2024-04-16T07:46:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33370.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33370.log.gz 2024-04-16T07:46:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32866.log 2024-04-16T07:46:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-16T07:46:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.30641.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30641.log.gz 2024-04-16T07:46:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31045.log 2024-04-16T07:46:52.167 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32866.log.gz 2024-04-16T07:46:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32380.log 2024-04-16T07:46:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.31045.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31045.log.gz 2024-04-16T07:46:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:gzip 82.5% -5 -- replaced with /var/log/ceph/ceph.log.gz --verbose 2024-04-16T07:46:52.167 INFO:teuthology.orchestra.run.smithi070.stderr: -- /var/log/ceph/ceph-client.admin.32379.log 2024-04-16T07:46:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32378.log 2024-04-16T07:46:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32380.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32380.log.gz 2024-04-16T07:46:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32379.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32379.log.gzgzip 2024-04-16T07:46:52.167 INFO:teuthology.orchestra.run.smithi070.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31428.log 2024-04-16T07:46:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30808.log 2024-04-16T07:46:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32378.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32378.log.gz 2024-04-16T07:46:52.168 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.31428.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31428.log.gz 2024-04-16T07:46:52.168 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32557.log 2024-04-16T07:46:52.168 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33286.log 2024-04-16T07:46:52.168 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.30808.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30808.log.gz 2024-04-16T07:46:52.168 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32557.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28754.log 2024-04-16T07:46:52.168 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32557.log.gz 2024-04-16T07:46:52.168 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30961.log 2024-04-16T07:46:52.168 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33286.log.gz 2024-04-16T07:46:52.168 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph.tmp-client.admin.28754.log: gzip -5 66.9% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28754.log.gz 2024-04-16T07:46:52.168 INFO:teuthology.orchestra.run.smithi070.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29900.log 2024-04-16T07:46:52.168 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.30961.log: gzip -5 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30961.log.gz 2024-04-16T07:46:52.168 INFO:teuthology.orchestra.run.smithi070.stderr: --verbose -- /var/log/ceph/ceph-client.admin.30541.log 2024-04-16T07:46:52.168 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.29900.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29900.log.gz 2024-04-16T07:46:52.168 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32992.log 2024-04-16T07:46:52.168 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5/var/log/ceph/ceph-client.admin.30541.log: --verbose -- /var/log/ceph/ceph-client.admin.33034.log 2024-04-16T07:46:52.168 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30541.log.gz 2024-04-16T07:46:52.168 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32992.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32992.log.gz 2024-04-16T07:46:52.169 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31331.log 2024-04-16T07:46:52.169 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33034.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33034.log.gz 2024-04-16T07:46:52.169 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.31331.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31331.log.gz 2024-04-16T07:46:52.183 INFO:teuthology.orchestra.run.smithi070.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-16T07:46:52.304 INFO:teuthology.orchestra.run.smithi070.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-16T07:46:52.335 INFO:teuthology.orchestra.run.smithi172.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-16T07:46:52.363 INFO:teuthology.orchestra.run.smithi172.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-16T07:46:52.365 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-04-16T07:46:52.365 INFO:teuthology.orchestra.run.smithi172.stderr:real 0m0.249s 2024-04-16T07:46:52.366 INFO:teuthology.orchestra.run.smithi172.stderr:user 0m0.450s 2024-04-16T07:46:52.366 INFO:teuthology.orchestra.run.smithi172.stderr:sys 0m0.033s 2024-04-16T07:46:52.371 INFO:teuthology.orchestra.run.smithi070.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-16T07:46:52.373 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-16T07:46:52.373 INFO:teuthology.orchestra.run.smithi070.stderr:real 0m0.259s 2024-04-16T07:46:52.374 INFO:teuthology.orchestra.run.smithi070.stderr:user 0m0.469s 2024-04-16T07:46:52.374 INFO:teuthology.orchestra.run.smithi070.stderr:sys 0m0.069s 2024-04-16T07:46:52.374 INFO:tasks.ceph:Archiving logs... 2024-04-16T07:46:52.374 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/log/ceph to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657976/remote/smithi070/log 2024-04-16T07:46:52.375 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T07:46:52.602 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657976/remote/smithi172/log 2024-04-16T07:46:52.603 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T07:46:52.736 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-16T07:46:52.752 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/rados.py", line 308, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/rados.py", line 289, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' 2024-04-16T07:46:52.752 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-16T07:46:52.753 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T07:46:52.755 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T07:46:52.847 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-16T07:46:52.856 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-16T07:46:52.856 DEBUG:teuthology.orchestra.run.smithi070:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T07:46:52.859 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T07:46:52.871 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-04-16T07:46:52.873 INFO:teuthology.orchestra.run.smithi172.stderr:bash: line 1: ntpq: command not found 2024-04-16T07:46:52.875 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T07:46:52.875 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-04-16T07:46:52.875 INFO:teuthology.orchestra.run.smithi070.stdout:^* hv01.front.sepia.ceph.com 3 6 377 53 +1650us[+1671us] +/- 54ms 2024-04-16T07:46:52.876 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 52 -504us[ -504us] +/- 49ms 2024-04-16T07:46:52.876 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 -1893us[-1893us] +/- 78ms 2024-04-16T07:46:52.876 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:46:52.877 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T07:46:52.877 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2024-04-16T07:46:52.877 INFO:teuthology.orchestra.run.smithi172.stdout:^* hv01.front.sepia.ceph.com 3 6 377 50 +1657us[+1652us] +/- 54ms 2024-04-16T07:46:52.877 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 52 -492us[ -498us] +/- 49ms 2024-04-16T07:46:52.878 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 -1878us[-1883us] +/- 78ms 2024-04-16T07:46:52.878 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:46:52.878 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-16T07:46:52.887 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-16T07:46:52.887 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-16T07:46:52.896 DEBUG:teuthology.orchestra.run.smithi070:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T07:46:52.982 DEBUG:teuthology.orchestra.run.smithi172:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T07:46:53.048 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T07:46:53.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:46:53.079 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T07:46:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:46:53.110 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-16T07:46:53.119 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-16T07:46:53.127 INFO:teuthology.task.internal:Duration was 837.997748 seconds 2024-04-16T07:46:53.127 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-16T07:46:53.138 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-16T07:46:53.138 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T07:46:53.140 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T07:46:53.183 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T07:46:53.195 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T07:46:53.368 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-16T07:46:53.368 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi070.front.sepia.ceph.com 2024-04-16T07:46:53.369 DEBUG:teuthology.orchestra.run.smithi070:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T07:46:53.388 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2024-04-16T07:46:53.388 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T07:46:53.417 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-16T07:46:53.417 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T07:46:53.431 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T07:46:53.487 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-16T07:46:53.487 DEBUG:teuthology.orchestra.run.smithi070:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T07:46:53.501 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T07:46:53.623 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-16T07:46:53.633 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-16T07:46:53.633 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T07:46:53.661 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T07:46:53.690 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-16T07:46:53.700 DEBUG:teuthology.orchestra.run.smithi070:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T07:46:53.704 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T07:46:53.728 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = core 2024-04-16T07:46:53.757 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2024-04-16T07:46:53.791 DEBUG:teuthology.orchestra.run.smithi070:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T07:46:53.809 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi070.front.sepia.ceph.com, flagging run as failed 2024-04-16T07:46:53.810 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T07:46:53.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:46:53.847 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-16T07:46:53.858 INFO:teuthology.task.internal:Transferring archived files... 2024-04-16T07:46:53.859 DEBUG:teuthology.misc:Transferring archived files from smithi070:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657976/remote/smithi070 2024-04-16T07:46:53.859 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T07:47:00.463 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2024-04-16T07:47:00.474 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657976/remote/smithi070/coredump/1713253569.31500.core.gz: gzip compressed data, was "1713253569.31500.core", last modified: Tue Apr 16 07:46:53 2024, max compression, original size modulo 2^32 207794176 2024-04-16T07:47:00.475 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2024-04-16T07:47:01.301 INFO:teuthology.task.internal: core looks like: /tmp/tmpefp7k77m: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-', real uid: 1000, effective uid: 1000, real gid: 1301, effective gid: 1301, execfn: '/usr/bin/ceph_test_rados', platform: 'x86_64' 2024-04-16T07:47:01.323 INFO:teuthology.task.internal: dump_program: ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in- 2024-04-16T07:47:01.323 DEBUG:teuthology.orchestra.run.smithi070:> which 'ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-' 2024-04-16T07:47:01.343 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/which: unrecognized option '--write-fadvise-dontneed' 2024-04-16T07:47:01.343 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/which: unrecognized option '--max-ops' 2024-04-16T07:47:01.343 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/which: unrecognized option '--objects' 2024-04-16T07:47:01.343 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/which: unrecognized option '--max-in-' 2024-04-16T07:47:01.343 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/which: no 4000 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2024-04-16T07:47:01.343 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/which: no 500 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2024-04-16T07:47:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-16T07:47:01.344 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph_test_rados 2024-04-16T07:47:01.345 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/rados.py", line 308, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/rados.py", line 289, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi070 with status 2: "which 'ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-'" 2024-04-16T07:47:01.346 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-16T07:47:01.358 INFO:teuthology.task.internal:Not uploading archives. 2024-04-16T07:47:01.358 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-16T07:47:01.399 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-16T07:47:01.400 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T07:47:01.403 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T07:47:01.418 INFO:teuthology.orchestra.run.smithi070.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 16 07:46 /home/ubuntu/cephtest 2024-04-16T07:47:01.419 INFO:teuthology.orchestra.run.smithi070.stdout: 262167 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Apr 16 07:46 /home/ubuntu/cephtest/archive 2024-04-16T07:47:01.419 INFO:teuthology.orchestra.run.smithi070.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 07:46 /home/ubuntu/cephtest/archive/syslog 2024-04-16T07:47:01.419 INFO:teuthology.orchestra.run.smithi070.stdout: 262223 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Apr 16 07:32 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-04-16T07:47:01.419 INFO:teuthology.orchestra.run.smithi070.stdout: 265306 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Apr 16 07:32 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-04-16T07:47:01.419 INFO:teuthology.orchestra.run.smithi070.stdout: 265305 180 -rw-r--r-- 1 ubuntu ubuntu 183638 Apr 16 07:46 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T07:47:01.419 INFO:teuthology.orchestra.run.smithi070.stdout: 403412 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 07:46 /home/ubuntu/cephtest/archive/audit 2024-04-16T07:47:01.419 INFO:teuthology.orchestra.run.smithi070.stdout: 403414 120 -rw------- 1 ubuntu ubuntu 122231 Apr 16 07:46 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-04-16T07:47:01.419 INFO:teuthology.orchestra.run.smithi070.stdout: 262168 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 07:46 /home/ubuntu/cephtest/archive/coredump 2024-04-16T07:47:01.419 INFO:teuthology.orchestra.run.smithi070.stdout: 267157 97012 -rw------- 1 ubuntu ubuntu 207794176 Apr 16 07:46 /home/ubuntu/cephtest/archive/coredump/1713253569.31500.core 2024-04-16T07:47:01.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:47:01.420 INFO:teuthology.orchestra.run.smithi070.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-16T07:47:01.420 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/rados.py", line 308, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/rados.py", line 289, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi070 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-16T07:47:01.421 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-16T07:47:01.432 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-16T07:47:01.443 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' 2024-04-16T07:47:01.443 INFO:teuthology.nuke:Checking targets against current locks 2024-04-16T07:47:01.447 INFO:teuthology.orchestra.run.smithi172.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 16 07:46 /home/ubuntu/cephtest 2024-04-16T07:47:01.447 INFO:teuthology.orchestra.run.smithi172.stdout: 262167 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Apr 16 07:46 /home/ubuntu/cephtest/archive 2024-04-16T07:47:01.447 INFO:teuthology.orchestra.run.smithi172.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 07:46 /home/ubuntu/cephtest/archive/syslog 2024-04-16T07:47:01.447 INFO:teuthology.orchestra.run.smithi172.stdout: 262223 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Apr 16 07:32 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-04-16T07:47:01.447 INFO:teuthology.orchestra.run.smithi172.stdout: 265306 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Apr 16 07:32 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-04-16T07:47:01.447 INFO:teuthology.orchestra.run.smithi172.stdout: 265305 180 -rw-r--r-- 1 ubuntu ubuntu 180438 Apr 16 07:46 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T07:47:01.447 INFO:teuthology.orchestra.run.smithi172.stdout: 403412 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 07:46 /home/ubuntu/cephtest/archive/audit 2024-04-16T07:47:01.447 INFO:teuthology.orchestra.run.smithi172.stdout: 403695 112 -rw------- 1 ubuntu ubuntu 112541 Apr 16 07:46 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-04-16T07:47:01.448 INFO:teuthology.orchestra.run.smithi172.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-16T07:47:01.468 DEBUG:teuthology.nuke:shortname: smithi070 2024-04-16T07:47:01.468 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T07:47:01.489 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657976', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 07:23:13.249089', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMsJ2TJdaZN5y4l3u/X2+t7XwFtINmipTx2SWpFb/+DHE7lIYCT/PxtPsat5G3qtz/BI6Ro1PNqxidKlyqL4WcI='} 2024-04-16T07:47:01.492 DEBUG:teuthology.nuke:shortname: smithi172 2024-04-16T07:47:01.493 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T07:47:01.516 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657976', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 07:23:13.250377', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMWPjIiC9oiD/C0llv5iLJ6LthJN1mBh2pA/Bpd0Ecmd+PM/Bw6gew0/xXSdD3v7vbwk6YRVobobOscRqkt5Ums='} 2024-04-16T07:47:01.533 INFO:teuthology.orchestra.console.smithi070:Power off 2024-04-16T07:47:01.533 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T07:47:01.558 DEBUG:teuthology.orchestra.console.smithi070:power off output: Chassis Power Control: Down/Off 2024-04-16T07:47:01.558 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T07:47:01.571 INFO:teuthology.orchestra.console.smithi172:Power off 2024-04-16T07:47:01.571 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T07:47:01.580 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2024-04-16T07:47:01.593 DEBUG:teuthology.orchestra.console.smithi172:power off output: Chassis Power Control: Down/Off 2024-04-16T07:47:01.593 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T07:47:01.615 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-04-16T07:47:05.581 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T07:47:05.616 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T07:47:05.694 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2024-04-16T07:47:05.725 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-04-16T07:47:09.698 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T07:47:09.726 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T07:47:09.810 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is off 2024-04-16T07:47:09.835 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is off 2024-04-16T07:47:09.911 INFO:teuthology.orchestra.console.smithi070:Power off completed 2024-04-16T07:47:09.936 INFO:teuthology.orchestra.console.smithi172:Power off completed 2024-04-16T07:47:10.041 INFO:teuthology.run:Summary data: description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} duration: 837.997748374939 failure_reason: 'Command crashed: ''CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0''' flavor: crimson owner: scheduled_matan@teuthology status: fail success: false 2024-04-16T07:47:10.041 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T07:47:10.124 INFO:teuthology.run:FAIL