2024-04-16T07:28:54.090 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-16T07:28:54.090 DEBUG:teuthology.run:Teuthology command: teuthology --description crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} --archive /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657957 --name matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi --owner scheduled_matan@teuthology --verbose -- /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657957/orig.config.yaml 2024-04-16T07:28:54.132 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T07:28:54.235 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657957 branch: wip-matanb-crimson-replicated-april-15 description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} email: null first_in_suite: false job_id: '7657957' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-replicated-april-15 ceph: conf: client.0: admin socket: /var/run/ceph/ceph-$name.asok global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: min_in: 4 workunit: branch: wip-matanb-crimson-replicated-april-15 env: CRIMSON_COMPAT: '1' sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 7594 sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-matanb-crimson-replicated-april-15 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBS07dOWcaLjUiOU6r19LBrr0T8Y6Zjb8ZCWZeDak9Zu/sx6QisBlA9o2b3LdPcm1uZCdsJBZZe7jXOMEGadN0c= smithi094.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCB2K1ya0mQ2pOgjC1b+JEgG9txeTVJWCARqu4FLJYKv0kBC4TWbRYstDVBDoII6KaXMqEibABpKWxNkhPZ2Des= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson - thrashosds: ceph_objectstore_tool: false chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - radosbench: clients: - client.0 time: 150 - admin_socket: client.0: objecter_requests: test: http://git.ceph.com/?p={repo};a=blob_plain;f=src/test/admin_socket/objecter_requests;hb={branch} teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-16_07:22:04 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-16T07:28:54.235 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa; will attempt to use it 2024-04-16T07:28:54.236 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks 2024-04-16T07:28:54.237 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-16T07:28:54.238 INFO:teuthology.task.internal:Checking packages... 2024-04-16T07:28:54.264 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '30b0aa0009a925d6a46536e55c5656edcd51ecdb' 2024-04-16T07:28:54.264 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T07:28:54.264 INFO:teuthology.packaging:ref: None 2024-04-16T07:28:54.264 INFO:teuthology.packaging:tag: None 2024-04-16T07:28:54.265 INFO:teuthology.packaging:branch: wip-matanb-crimson-replicated-april-15 2024-04-16T07:28:54.265 INFO:teuthology.packaging:sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:28:54.265 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-replicated-april-15 2024-04-16T07:28:54.484 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3109.g30b0aa00 2024-04-16T07:28:54.485 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-16T07:28:54.494 INFO:teuthology.task.internal:no buildpackages task found 2024-04-16T07:28:54.494 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-16T07:28:54.541 INFO:teuthology.task.internal:Saving configuration 2024-04-16T07:28:54.556 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-16T07:28:54.570 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T07:28:54.593 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657957', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 07:23:06.314856', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBS07dOWcaLjUiOU6r19LBrr0T8Y6Zjb8ZCWZeDak9Zu/sx6QisBlA9o2b3LdPcm1uZCdsJBZZe7jXOMEGadN0c='} 2024-04-16T07:28:54.618 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657957', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 07:23:06.316279', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCB2K1ya0mQ2pOgjC1b+JEgG9txeTVJWCARqu4FLJYKv0kBC4TWbRYstDVBDoII6KaXMqEibABpKWxNkhPZ2Des='} 2024-04-16T07:28:54.618 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-16T07:28:54.627 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-04-16T07:28:54.627 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-04-16T07:28:54.627 INFO:teuthology.run_tasks:Running task console_log... 2024-04-16T07:28:54.725 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9b8519fe50>, signals=[15]) 2024-04-16T07:28:54.725 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-16T07:28:54.736 INFO:teuthology.task.internal:Opening connections... 2024-04-16T07:28:54.737 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2024-04-16T07:28:54.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:28:54.845 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi094.front.sepia.ceph.com 2024-04-16T07:28:54.846 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:28:54.932 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-16T07:28:54.941 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2024-04-16T07:28:54.959 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2024-04-16T07:28:54.959 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2024-04-16T07:28:55.016 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2024-04-16T07:28:55.016 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9" 2024-04-16T07:28:55.016 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2024-04-16T07:28:55.016 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2024-04-16T07:28:55.016 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9" 2024-04-16T07:28:55.016 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9" 2024-04-16T07:28:55.016 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T07:28:55.016 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2024-04-16T07:28:55.016 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon" 2024-04-16T07:28:55.017 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T07:28:55.017 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2024-04-16T07:28:55.017 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T07:28:55.017 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T07:28:55.017 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T07:28:55.017 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2024-04-16T07:28:55.039 DEBUG:teuthology.orchestra.run.smithi094:> uname -m 2024-04-16T07:28:55.056 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2024-04-16T07:28:55.056 DEBUG:teuthology.orchestra.run.smithi094:> cat /etc/os-release 2024-04-16T07:28:55.116 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="CentOS Stream" 2024-04-16T07:28:55.116 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="9" 2024-04-16T07:28:55.116 INFO:teuthology.orchestra.run.smithi094.stdout:ID="centos" 2024-04-16T07:28:55.116 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE="rhel fedora" 2024-04-16T07:28:55.116 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="9" 2024-04-16T07:28:55.117 INFO:teuthology.orchestra.run.smithi094.stdout:PLATFORM_ID="platform:el9" 2024-04-16T07:28:55.117 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T07:28:55.117 INFO:teuthology.orchestra.run.smithi094.stdout:ANSI_COLOR="0;31" 2024-04-16T07:28:55.117 INFO:teuthology.orchestra.run.smithi094.stdout:LOGO="fedora-logo-icon" 2024-04-16T07:28:55.117 INFO:teuthology.orchestra.run.smithi094.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T07:28:55.117 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://centos.org/" 2024-04-16T07:28:55.117 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T07:28:55.117 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T07:28:55.117 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T07:28:55.117 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2024-04-16T07:28:55.139 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-16T07:28:55.151 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-16T07:28:55.175 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-16T07:28:55.175 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2024-04-16T07:28:55.177 DEBUG:teuthology.orchestra.run.smithi094:> test '!' -e /home/ubuntu/cephtest 2024-04-16T07:28:55.193 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-16T07:28:55.204 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-16T07:28:55.204 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2024-04-16T07:28:55.234 DEBUG:teuthology.orchestra.run.smithi094:> test -z $(ls -A /var/lib/ceph) 2024-04-16T07:28:55.262 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-16T07:28:55.317 INFO:teuthology.run_tasks:Running task kernel... 2024-04-16T07:28:55.335 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-16T07:28:55.335 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-16T07:28:55.335 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T07:28:55.336 DEBUG:teuthology.orchestra.run.smithi094:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T07:28:55.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:28:55.351 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-04-16T07:28:55.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:28:55.352 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-04-16T07:28:55.412 INFO:teuthology.orchestra.run.smithi094.stdout:5.14.0-361.el9.x86_64 2024-04-16T07:28:55.412 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-361.el9.x86_64 2024-04-16T07:28:55.412 INFO:teuthology.task.kernel:Running kernel on smithi094: 5.14.0-361.el9.x86_64 2024-04-16T07:28:55.412 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum install -y kernel 2024-04-16T07:28:55.413 INFO:teuthology.task.kernel:Running kernel on smithi031: 5.14.0-361.el9.x86_64 2024-04-16T07:28:55.413 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel 2024-04-16T07:28:57.494 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-04-16T07:28:57.505 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-04-16T07:28:58.919 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-04-16T07:28:58.938 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-04-16T07:29:00.196 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 8.8 MB/s | 8.0 MB 00:00 2024-04-16T07:29:00.660 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.0 MB 00:01 2024-04-16T07:29:02.595 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-04-16T07:29:04.346 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 6.3 MB/s | 19 MB 00:02 2024-04-16T07:29:07.746 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.2 MB 00:01 2024-04-16T07:29:09.222 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 6.2 MB 00:01 2024-04-16T07:29:09.230 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-04-16T07:29:10.233 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-04-16T07:29:10.947 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-04-16T07:29:12.132 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-04-16T07:29:15.274 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-04-16T07:29:17.374 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-16T07:29:17.656 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-16T07:29:17.656 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-16T07:29:17.753 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:29:17.763 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:29:17.763 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-04-16T07:29:17.763 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:29:17.764 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-16T07:29:17.764 INFO:teuthology.orchestra.run.smithi094.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-16T07:29:17.764 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-04-16T07:29:17.764 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-16T07:29:17.764 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-16T07:29:17.764 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-16T07:29:17.764 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:29:17.764 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-16T07:29:17.764 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:29:17.764 INFO:teuthology.orchestra.run.smithi094.stdout:Install 4 Packages 2024-04-16T07:29:17.764 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:29:17.765 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 77 M 2024-04-16T07:29:17.765 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 123 M 2024-04-16T07:29:17.765 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-16T07:29:18.053 INFO:teuthology.orchestra.run.smithi094.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 964 kB/s | 277 kB 00:00 2024-04-16T07:29:18.978 INFO:teuthology.orchestra.run.smithi094.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 13 MB/s | 15 MB 00:01 2024-04-16T07:29:19.695 INFO:teuthology.orchestra.run.smithi094.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 17 MB/s | 34 MB 00:01 2024-04-16T07:29:19.867 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-16T07:29:19.867 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-16T07:29:19.960 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:29:19.970 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:29:19.970 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-16T07:29:19.970 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:29:19.970 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-16T07:29:19.970 INFO:teuthology.orchestra.run.smithi031.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-16T07:29:19.970 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-16T07:29:19.970 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-16T07:29:19.971 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-16T07:29:19.971 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-16T07:29:19.971 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:29:19.971 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-16T07:29:19.971 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:29:19.971 INFO:teuthology.orchestra.run.smithi031.stdout:Install 4 Packages 2024-04-16T07:29:19.971 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:29:19.971 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 77 M 2024-04-16T07:29:19.972 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 123 M 2024-04-16T07:29:19.972 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-16T07:29:20.045 INFO:teuthology.orchestra.run.smithi094.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 14 MB/s | 28 MB 00:01 2024-04-16T07:29:20.048 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:29:20.048 INFO:teuthology.orchestra.run.smithi094.stdout:Total 34 MB/s | 77 MB 00:02 2024-04-16T07:29:20.049 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-16T07:29:20.150 INFO:teuthology.orchestra.run.smithi031.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.5 MB/s | 277 kB 00:00 2024-04-16T07:29:20.412 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-16T07:29:20.412 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-16T07:29:20.717 INFO:teuthology.orchestra.run.smithi031.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-04-16T07:29:21.235 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-16T07:29:21.235 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-16T07:29:21.317 INFO:teuthology.orchestra.run.smithi031.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 25 MB/s | 34 MB 00:01 2024-04-16T07:29:21.892 INFO:teuthology.orchestra.run.smithi031.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 16 MB/s | 28 MB 00:01 2024-04-16T07:29:21.895 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:29:21.895 INFO:teuthology.orchestra.run.smithi031.stdout:Total 40 MB/s | 77 MB 00:01 2024-04-16T07:29:21.895 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-16T07:29:22.336 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-16T07:29:22.336 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-16T07:29:23.210 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-16T07:29:23.210 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-16T07:29:23.514 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-16T07:29:24.149 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-16T07:29:24.232 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:29:25.646 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-16T07:29:26.261 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-16T07:29:26.351 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:29:26.994 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:29:27.140 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:29:29.229 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:29:29.367 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:29:31.830 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:29:31.864 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:29:34.466 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:29:34.508 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:29:36.656 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:29:39.634 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:06.194 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:06.194 INFO:teuthology.orchestra.run.smithi094.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T07:30:06.194 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:30:06.237 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-16T07:30:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:30:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:30:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:30:07.039 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-16T07:30:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-16T07:30:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-16T07:30:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-16T07:30:07.040 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-16T07:30:07.040 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:30:07.040 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:30:07.391 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T07:30:07.487 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T07:30:07.488 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T07:30:07.488 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-16T07:30:07.488 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-16T07:30:07.488 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-16T07:30:07.488 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-04-16T07:30:07.488 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi094.front.sepia.ceph.com, path=None, version=distro) 2024-04-16T07:30:07.488 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T07:30:07.568 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T07:30:07.569 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T07:30:07.569 DEBUG:teuthology.orchestra.run.smithi094:> sudo rpm -qi grub2-tools 2024-04-16T07:30:07.611 INFO:teuthology.orchestra.run.smithi094.stdout:Name : grub2-tools 2024-04-16T07:30:07.611 INFO:teuthology.orchestra.run.smithi094.stdout:Epoch : 1 2024-04-16T07:30:07.611 INFO:teuthology.orchestra.run.smithi094.stdout:Version : 2.06 2024-04-16T07:30:07.611 INFO:teuthology.orchestra.run.smithi094.stdout:Release : 68.el9 2024-04-16T07:30:07.611 INFO:teuthology.orchestra.run.smithi094.stdout:Architecture: x86_64 2024-04-16T07:30:07.611 INFO:teuthology.orchestra.run.smithi094.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T07:30:07.611 INFO:teuthology.orchestra.run.smithi094.stdout:Group : Unspecified 2024-04-16T07:30:07.611 INFO:teuthology.orchestra.run.smithi094.stdout:Size : 8264254 2024-04-16T07:30:07.611 INFO:teuthology.orchestra.run.smithi094.stdout:License : GPLv3+ 2024-04-16T07:30:07.611 INFO:teuthology.orchestra.run.smithi094.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T07:30:07.611 INFO:teuthology.orchestra.run.smithi094.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T07:30:07.611 INFO:teuthology.orchestra.run.smithi094.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T07:30:07.612 INFO:teuthology.orchestra.run.smithi094.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T07:30:07.612 INFO:teuthology.orchestra.run.smithi094.stdout:Packager : builder@centos.org 2024-04-16T07:30:07.612 INFO:teuthology.orchestra.run.smithi094.stdout:Vendor : CentOS 2024-04-16T07:30:07.612 INFO:teuthology.orchestra.run.smithi094.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T07:30:07.612 INFO:teuthology.orchestra.run.smithi094.stdout:Summary : Support tools for GRUB. 2024-04-16T07:30:07.612 INFO:teuthology.orchestra.run.smithi094.stdout:Description : 2024-04-16T07:30:07.612 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:30:07.612 INFO:teuthology.orchestra.run.smithi094.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T07:30:07.612 INFO:teuthology.orchestra.run.smithi094.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T07:30:07.612 INFO:teuthology.orchestra.run.smithi094.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T07:30:07.612 INFO:teuthology.orchestra.run.smithi094.stdout:hardware devices. 2024-04-16T07:30:07.612 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:30:07.612 INFO:teuthology.orchestra.run.smithi094.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T07:30:07.613 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T07:30:07.613 INFO:teuthology.task.kernel:Updating grub on smithi094 to boot 5.14.0-438.el9.x86_64 2024-04-16T07:30:07.613 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T07:30:08.302 INFO:teuthology.orchestra.run.smithi094.stderr:Generating grub configuration file ... 2024-04-16T07:30:09.705 INFO:teuthology.orchestra.run.smithi094.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T07:30:09.754 INFO:teuthology.orchestra.run.smithi094.stderr:done 2024-04-16T07:30:09.757 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-04-16T07:30:09.771 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.RaUQ0PJQxH 2024-04-16T07:30:09.772 DEBUG:teuthology.orchestra.run.smithi094:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RaUQ0PJQxH 2024-04-16T07:30:09.840 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.RaUQ0PJQxH 2024-04-16T07:30:09.974 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.RaUQ0PJQxH is 6KB 2024-04-16T07:30:10.022 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.RaUQ0PJQxH 2024-04-16T07:30:10.037 DEBUG:teuthology.orchestra.run.smithi094:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T07:30:10.107 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T07:30:10.108 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T07:30:10.108 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T07:30:10.108 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-16T07:30:10.109 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-16T07:30:10.263 DEBUG:teuthology.orchestra.run.smithi094:> sudo shutdown -r now 2024-04-16T07:30:10.816 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:10.816 INFO:teuthology.orchestra.run.smithi031.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T07:30:10.816 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:30:10.861 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:11.242 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:11.242 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-16T07:30:11.242 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:30:11.242 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:30:11.662 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:11.662 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:30:11.662 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-16T07:30:11.662 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-16T07:30:11.662 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-16T07:30:11.662 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-16T07:30:11.663 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-16T07:30:11.663 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:30:11.663 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:30:12.012 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T07:30:12.114 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T07:30:12.114 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T07:30:12.114 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-16T07:30:12.115 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-16T07:30:12.115 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-16T07:30:12.115 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-16T07:30:12.115 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi031.front.sepia.ceph.com, path=None, version=distro) 2024-04-16T07:30:12.115 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T07:30:12.199 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T07:30:12.199 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T07:30:12.200 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -qi grub2-tools 2024-04-16T07:30:12.237 INFO:teuthology.orchestra.run.smithi031.stdout:Name : grub2-tools 2024-04-16T07:30:12.237 INFO:teuthology.orchestra.run.smithi031.stdout:Epoch : 1 2024-04-16T07:30:12.237 INFO:teuthology.orchestra.run.smithi031.stdout:Version : 2.06 2024-04-16T07:30:12.237 INFO:teuthology.orchestra.run.smithi031.stdout:Release : 68.el9 2024-04-16T07:30:12.237 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: x86_64 2024-04-16T07:30:12.237 INFO:teuthology.orchestra.run.smithi031.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T07:30:12.237 INFO:teuthology.orchestra.run.smithi031.stdout:Group : Unspecified 2024-04-16T07:30:12.238 INFO:teuthology.orchestra.run.smithi031.stdout:Size : 8264254 2024-04-16T07:30:12.238 INFO:teuthology.orchestra.run.smithi031.stdout:License : GPLv3+ 2024-04-16T07:30:12.238 INFO:teuthology.orchestra.run.smithi031.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T07:30:12.238 INFO:teuthology.orchestra.run.smithi031.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T07:30:12.238 INFO:teuthology.orchestra.run.smithi031.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T07:30:12.238 INFO:teuthology.orchestra.run.smithi031.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T07:30:12.238 INFO:teuthology.orchestra.run.smithi031.stdout:Packager : builder@centos.org 2024-04-16T07:30:12.238 INFO:teuthology.orchestra.run.smithi031.stdout:Vendor : CentOS 2024-04-16T07:30:12.238 INFO:teuthology.orchestra.run.smithi031.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T07:30:12.238 INFO:teuthology.orchestra.run.smithi031.stdout:Summary : Support tools for GRUB. 2024-04-16T07:30:12.238 INFO:teuthology.orchestra.run.smithi031.stdout:Description : 2024-04-16T07:30:12.238 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:30:12.238 INFO:teuthology.orchestra.run.smithi031.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T07:30:12.238 INFO:teuthology.orchestra.run.smithi031.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T07:30:12.239 INFO:teuthology.orchestra.run.smithi031.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T07:30:12.239 INFO:teuthology.orchestra.run.smithi031.stdout:hardware devices. 2024-04-16T07:30:12.239 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:30:12.239 INFO:teuthology.orchestra.run.smithi031.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T07:30:12.239 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T07:30:12.240 INFO:teuthology.task.kernel:Updating grub on smithi031 to boot 5.14.0-438.el9.x86_64 2024-04-16T07:30:12.240 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T07:30:12.885 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2024-04-16T07:30:14.204 INFO:teuthology.orchestra.run.smithi031.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T07:30:14.240 INFO:teuthology.orchestra.run.smithi031.stderr:done 2024-04-16T07:30:14.242 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-04-16T07:30:14.257 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.hQLMtMJX2o 2024-04-16T07:30:14.257 DEBUG:teuthology.orchestra.run.smithi031:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.hQLMtMJX2o 2024-04-16T07:30:14.333 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.hQLMtMJX2o 2024-04-16T07:30:14.476 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.hQLMtMJX2o is 6KB 2024-04-16T07:30:14.525 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.hQLMtMJX2o 2024-04-16T07:30:14.540 DEBUG:teuthology.orchestra.run.smithi031:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T07:30:14.608 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T07:30:14.608 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T07:30:14.608 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T07:30:14.608 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-16T07:30:14.610 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-16T07:30:14.754 DEBUG:teuthology.orchestra.run.smithi031:> sudo shutdown -r now 2024-04-16T07:30:40.291 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-04-16T07:30:40.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-04-16T07:30:40.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:30:44.761 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-16T07:30:44.761 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-16T07:30:44.762 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:31:03.326 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-04-16T07:31:12.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-16T07:31:12.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:31:30.718 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-04-16T07:31:40.304 DEBUG:teuthology.orchestra.remote:timed out 2024-04-16T07:31:42.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-16T07:31:42.723 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:31:43.158 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-04-16T07:31:43.562 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-16T07:31:43.562 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-438.el9.x86_64"... 2024-04-16T07:31:43.563 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-04-16T07:31:43.579 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-438.el9.x86_64 2024-04-16T07:31:43.579 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-16T07:31:43.579 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T07:31:43.579 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T07:31:44.580 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-16T07:31:44.581 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T07:31:44.639 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2024-04-16T07:31:44.677 DEBUG:teuthology.parallel:result is None 2024-04-16T07:31:49.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-04-16T07:31:49.306 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:31:52.861 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-04-16T07:31:53.278 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-04-16T07:31:53.278 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-438.el9.x86_64"... 2024-04-16T07:31:53.278 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-04-16T07:31:53.293 INFO:teuthology.orchestra.run.smithi094.stdout:5.14.0-438.el9.x86_64 2024-04-16T07:31:53.294 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-16T07:31:53.294 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T07:31:53.294 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T07:31:54.295 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-04-16T07:31:54.295 DEBUG:teuthology.orchestra.run.smithi094:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T07:31:54.367 INFO:teuthology.orchestra.run.smithi094.stdout:ttyS1 2024-04-16T07:31:54.409 DEBUG:teuthology.parallel:result is None 2024-04-16T07:31:54.410 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-16T07:31:54.418 INFO:teuthology.task.internal:Creating test directory... 2024-04-16T07:31:54.418 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T07:31:54.422 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T07:31:54.439 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-16T07:31:54.445 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-16T07:31:54.474 INFO:teuthology.task.internal:Creating archive directory... 2024-04-16T07:31:54.474 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T07:31:54.482 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T07:31:54.519 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-16T07:31:54.525 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-16T07:31:54.526 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T07:31:54.547 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T07:31:54.589 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:31:54.593 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:31:54.604 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:31:54.607 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:31:54.608 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-16T07:31:54.615 INFO:teuthology.task.internal:Configuring sudo... 2024-04-16T07:31:54.615 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T07:31:54.650 DEBUG:teuthology.orchestra.run.smithi094:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T07:31:54.678 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-16T07:31:54.688 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-16T07:31:54.688 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T07:31:54.721 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T07:31:54.736 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:31:54.820 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:31:54.870 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:31:54.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:31:54.998 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-16T07:31:54.998 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T07:31:55.067 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:31:55.109 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:31:55.157 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:31:55.239 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:31:55.283 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-16T07:31:55.283 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T07:31:55.354 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2024-04-16T07:31:55.357 DEBUG:teuthology.orchestra.run.smithi094:> sudo service rsyslog restart 2024-04-16T07:31:55.408 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T07:31:55.450 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T07:31:55.711 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-16T07:31:55.718 INFO:teuthology.task.internal:Starting timer... 2024-04-16T07:31:55.718 INFO:teuthology.run_tasks:Running task pcp... 2024-04-16T07:31:55.745 INFO:teuthology.run_tasks:Running task selinux... 2024-04-16T07:31:55.777 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-16T07:31:55.778 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2024-04-16T07:31:55.841 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2024-04-16T07:31:55.843 DEBUG:teuthology.orchestra.run.smithi094:> sudo service auditd rotate 2024-04-16T07:31:55.909 INFO:teuthology.orchestra.run.smithi094.stdout:Rotating logs: 2024-04-16T07:31:55.911 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-16T07:31:55.911 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2024-04-16T07:31:55.931 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2024-04-16T07:31:55.931 DEBUG:teuthology.orchestra.run.smithi094:> /usr/sbin/getenforce 2024-04-16T07:31:55.952 INFO:teuthology.orchestra.run.smithi094.stdout:Permissive 2024-04-16T07:31:55.952 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi094.front.sepia.ceph.com': 'permissive'} 2024-04-16T07:31:55.953 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T07:31:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:31:56.002 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T07:31:56.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:31:56.030 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-16T07:31:56.030 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2024-04-16T07:31:56.075 DEBUG:teuthology.orchestra.run.smithi094:> sudo /usr/sbin/setenforce permissive 2024-04-16T07:31:56.112 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-16T07:31:56.122 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-16T07:31:56.137 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-16T07:31:56.139 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com,smithi094.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-16T07:36:18.730 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com'), Remote(name='ubuntu@smithi094.front.sepia.ceph.com')] 2024-04-16T07:36:18.732 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-16T07:36:18.734 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:36:18.849 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-04-16T07:36:18.940 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-16T07:36:18.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-04-16T07:36:18.941 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:36:19.062 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-04-16T07:36:19.147 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-04-16T07:36:19.148 INFO:teuthology.run_tasks:Running task clock... 2024-04-16T07:36:19.162 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-16T07:36:19.162 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T07:36:19.163 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T07:36:19.165 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T07:36:19.166 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T07:36:19.202 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T07:36:19.220 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T07:36:19.222 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T07:36:19.241 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T07:36:19.255 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2024-04-16T07:36:19.269 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2024-04-16T07:36:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:sudo: ntpd: command not found 2024-04-16T07:36:19.287 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T07:36:19.288 INFO:teuthology.orchestra.run.smithi094.stdout:506 Cannot talk to daemon 2024-04-16T07:36:19.307 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T07:36:19.307 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T07:36:19.323 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T07:36:19.369 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-04-16T07:36:19.373 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T07:36:19.373 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-04-16T07:36:19.373 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:36:19.373 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:36:19.373 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:36:19.373 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:36:19.379 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-04-16T07:36:19.383 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T07:36:19.383 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-04-16T07:36:19.384 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:36:19.384 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:36:19.384 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:36:19.384 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:36:19.384 INFO:teuthology.run_tasks:Running task install... 2024-04-16T07:36:19.393 DEBUG:teuthology.task.install:project ceph 2024-04-16T07:36:19.393 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}} 2024-04-16T07:36:19.393 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'} 2024-04-16T07:36:19.393 INFO:teuthology.task.install:Using flavor: crimson 2024-04-16T07:36:19.400 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-16T07:36:19.400 INFO:teuthology.task.install:extra packages: [] 2024-04-16T07:36:19.400 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'tag': None, 'wait_for_package': False} 2024-04-16T07:36:19.401 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:36:19.404 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'tag': None, 'wait_for_package': False} 2024-04-16T07:36:19.405 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:36:19.556 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/ 2024-04-16T07:36:19.556 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3109.g30b0aa00 2024-04-16T07:36:19.631 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/ 2024-04-16T07:36:19.631 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3109.g30b0aa00 2024-04-16T07:36:19.669 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T07:36:19.670 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-16T07:36:19.670 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T07:36:19.732 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T07:36:19.732 DEBUG:teuthology.orchestra.run.smithi094:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/30b0aa0009a925d6a46536e55c5656edcd51ecdb/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T07:36:19.802 DEBUG:teuthology.orchestra.run.smithi094:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T07:36:19.885 DEBUG:teuthology.orchestra.run.smithi094:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T07:36:19.953 INFO:teuthology.orchestra.run.smithi094.stdout:check_obsoletes = 1 2024-04-16T07:36:19.955 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum clean all 2024-04-16T07:36:19.976 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T07:36:19.977 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-16T07:36:19.977 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T07:36:20.010 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T07:36:20.010 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/30b0aa0009a925d6a46536e55c5656edcd51ecdb/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T07:36:20.083 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T07:36:20.169 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T07:36:20.194 INFO:teuthology.orchestra.run.smithi094.stdout:74 files removed 2024-04-16T07:36:20.200 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2024-04-16T07:36:20.203 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2024-04-16T07:36:20.223 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-radosgw 2024-04-16T07:36:20.452 INFO:teuthology.orchestra.run.smithi031.stdout:74 files removed 2024-04-16T07:36:20.480 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw 2024-04-16T07:36:21.057 INFO:teuthology.orchestra.run.smithi094.stdout:ceph packages for x86_64 165 kB/s | 87 kB 00:00 2024-04-16T07:36:21.175 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 226 kB/s | 87 kB 00:00 2024-04-16T07:36:21.508 INFO:teuthology.orchestra.run.smithi094.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-04-16T07:36:21.559 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-04-16T07:36:21.927 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 6.2 kB/s | 2.0 kB 00:00 2024-04-16T07:36:22.250 INFO:teuthology.orchestra.run.smithi094.stdout:ceph source packages 2.9 kB/s | 2.0 kB 00:00 2024-04-16T07:36:22.562 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-04-16T07:36:22.735 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-04-16T07:36:23.996 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-04-16T07:36:24.344 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-04-16T07:36:26.618 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-04-16T07:36:27.253 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.0 MB 00:02 2024-04-16T07:36:29.554 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-04-16T07:36:30.352 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 7.2 MB/s | 19 MB 00:02 2024-04-16T07:36:35.355 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.2 MB 00:01 2024-04-16T07:36:36.806 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 16 kB 00:00 2024-04-16T07:36:38.597 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-16T07:36:41.079 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 808 kB/s | 6.2 MB 00:07 2024-04-16T07:36:42.563 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 16 kB 00:00 2024-04-16T07:36:43.647 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-16T07:36:44.139 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-04-16T07:36:46.402 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:36:46.418 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:36:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-04-16T07:36:46.419 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:36:46.419 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-16T07:36:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 10 M 2024-04-16T07:36:46.419 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrading: 2024-04-16T07:36:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: librados2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.4 M 2024-04-16T07:36:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.7 M 2024-04-16T07:36:46.419 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-04-16T07:36:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options 2024-04-16T07:36:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T07:36:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.9 M 2024-04-16T07:36:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 19 M 2024-04-16T07:36:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 25 k 2024-04-16T07:36:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T07:36:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T07:36:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T07:36:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 797 k 2024-04-16T07:36:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T07:36:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T07:36:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T07:36:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 546 k 2024-04-16T07:36:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T07:36:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.2 M 2024-04-16T07:36:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T07:36:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T07:36:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T07:36:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse 2024-04-16T07:36:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 45 k 2024-04-16T07:36:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 135 k 2024-04-16T07:36:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 159 k 2024-04-16T07:36:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T07:36:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 315 k 2024-04-16T07:36:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 293 k 2024-04-16T07:36:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 99 k 2024-04-16T07:36:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-16T07:36:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-16T07:36:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T07:36:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T07:36:46.421 INFO:teuthology.orchestra.run.smithi094.stdout:Installing weak dependencies: 2024-04-16T07:36:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-16T07:36:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:36:46.421 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-16T07:36:46.422 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:36:46.422 INFO:teuthology.orchestra.run.smithi094.stdout:Install 30 Packages 2024-04-16T07:36:46.422 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrade 2 Packages 2024-04-16T07:36:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:36:46.422 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 59 M 2024-04-16T07:36:46.422 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-16T07:36:48.915 INFO:teuthology.orchestra.run.smithi094.stdout:(1/32): ceph-common-19.0.0-3109.g30b0aa00.el9.x 8.3 MB/s | 19 MB 00:02 2024-04-16T07:36:48.973 INFO:teuthology.orchestra.run.smithi094.stdout:(2/32): ceph-selinux-19.0.0-3109.g30b0aa00.el9. 434 kB/s | 25 kB 00:00 2024-04-16T07:36:49.132 INFO:teuthology.orchestra.run.smithi094.stdout:(3/32): libcephfs2-19.0.0-3109.g30b0aa00.el9.x8 4.9 MB/s | 797 kB 00:00 2024-04-16T07:36:49.232 INFO:teuthology.orchestra.run.smithi094.stdout:(4/32): libradosstriper1-19.0.0-3109.g30b0aa00. 5.4 MB/s | 546 kB 00:00 2024-04-16T07:36:49.464 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-04-16T07:36:49.816 INFO:teuthology.orchestra.run.smithi094.stdout:(5/32): librgw2-19.0.0-3109.g30b0aa00.el9.x86_6 9.0 MB/s | 5.2 MB 00:00 2024-04-16T07:36:49.858 INFO:teuthology.orchestra.run.smithi094.stdout:(6/32): python3-ceph-argparse-19.0.0-3109.g30b0 1.1 MB/s | 45 kB 00:00 2024-04-16T07:36:49.908 INFO:teuthology.orchestra.run.smithi094.stdout:(7/32): python3-ceph-common-19.0.0-3109.g30b0aa 2.6 MB/s | 135 kB 00:00 2024-04-16T07:36:49.966 INFO:teuthology.orchestra.run.smithi094.stdout:(8/32): python3-cephfs-19.0.0-3109.g30b0aa00.el 2.7 MB/s | 159 kB 00:00 2024-04-16T07:36:50.017 INFO:teuthology.orchestra.run.smithi094.stdout:(9/32): python3-rados-19.0.0-3109.g30b0aa00.el9 6.2 MB/s | 315 kB 00:00 2024-04-16T07:36:50.067 INFO:teuthology.orchestra.run.smithi094.stdout:(10/32): python3-rbd-19.0.0-3109.g30b0aa00.el9. 5.7 MB/s | 293 kB 00:00 2024-04-16T07:36:50.117 INFO:teuthology.orchestra.run.smithi094.stdout:(11/32): python3-rgw-19.0.0-3109.g30b0aa00.el9. 1.9 MB/s | 99 kB 00:00 2024-04-16T07:36:50.242 INFO:teuthology.orchestra.run.smithi094.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 848 kB/s | 106 kB 00:00 2024-04-16T07:36:50.293 INFO:teuthology.orchestra.run.smithi094.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-04-16T07:36:50.326 INFO:teuthology.orchestra.run.smithi094.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-04-16T07:36:50.396 INFO:teuthology.orchestra.run.smithi094.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-04-16T07:36:50.435 INFO:teuthology.orchestra.run.smithi094.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-04-16T07:36:50.469 INFO:teuthology.orchestra.run.smithi094.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-04-16T07:36:50.511 INFO:teuthology.orchestra.run.smithi094.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2024-04-16T07:36:50.544 INFO:teuthology.orchestra.run.smithi094.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 35 kB 00:00 2024-04-16T07:36:50.586 INFO:teuthology.orchestra.run.smithi094.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-04-16T07:36:50.689 INFO:teuthology.orchestra.run.smithi094.stdout:(21/32): ceph-radosgw-19.0.0-3109.g30b0aa00.el9 2.5 MB/s | 10 MB 00:04 2024-04-16T07:36:50.904 INFO:teuthology.orchestra.run.smithi094.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 975 kB/s | 309 kB 00:00 2024-04-16T07:36:50.962 INFO:teuthology.orchestra.run.smithi094.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-04-16T07:36:51.029 INFO:teuthology.orchestra.run.smithi094.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 731 kB/s | 49 kB 00:00 2024-04-16T07:36:51.146 INFO:teuthology.orchestra.run.smithi094.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 9.6 MB/s | 4.4 MB 00:00 2024-04-16T07:36:51.171 INFO:teuthology.orchestra.run.smithi094.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 474 kB/s | 67 kB 00:00 2024-04-16T07:36:51.214 INFO:teuthology.orchestra.run.smithi094.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-04-16T07:36:51.241 INFO:teuthology.orchestra.run.smithi094.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-04-16T07:36:51.289 INFO:teuthology.orchestra.run.smithi094.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-04-16T07:36:51.398 INFO:teuthology.orchestra.run.smithi094.stdout:(30/32): ceph-base-19.0.0-3109.g30b0aa00.el9.x8 1.2 MB/s | 5.9 MB 00:04 2024-04-16T07:36:51.540 INFO:teuthology.orchestra.run.smithi094.stdout:(31/32): librados2-19.0.0-3109.g30b0aa00.el9.x8 11 MB/s | 3.4 MB 00:00 2024-04-16T07:36:52.056 INFO:teuthology.orchestra.run.smithi094.stdout:(32/32): librbd1-19.0.0-3109.g30b0aa00.el9.x86_ 4.8 MB/s | 3.7 MB 00:00 2024-04-16T07:36:52.059 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:36:52.060 INFO:teuthology.orchestra.run.smithi094.stdout:Total 10 MB/s | 59 MB 00:05 2024-04-16T07:36:52.060 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-16T07:36:52.099 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-16T07:36:52.099 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-16T07:36:52.347 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:36:52.354 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:36:52.354 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-16T07:36:52.355 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:36:52.355 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-16T07:36:52.355 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 10 M 2024-04-16T07:36:52.355 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2024-04-16T07:36:52.355 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.4 M 2024-04-16T07:36:52.355 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.7 M 2024-04-16T07:36:52.356 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-16T07:36:52.356 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options 2024-04-16T07:36:52.356 INFO:teuthology.orchestra.run.smithi031.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T07:36:52.356 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.9 M 2024-04-16T07:36:52.356 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 19 M 2024-04-16T07:36:52.356 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 25 k 2024-04-16T07:36:52.357 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T07:36:52.357 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T07:36:52.357 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T07:36:52.357 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 797 k 2024-04-16T07:36:52.357 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T07:36:52.357 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T07:36:52.357 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T07:36:52.358 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 546 k 2024-04-16T07:36:52.358 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T07:36:52.358 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.2 M 2024-04-16T07:36:52.358 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T07:36:52.358 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T07:36:52.358 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T07:36:52.359 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse 2024-04-16T07:36:52.359 INFO:teuthology.orchestra.run.smithi031.stdout: x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 45 k 2024-04-16T07:36:52.359 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 135 k 2024-04-16T07:36:52.359 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 159 k 2024-04-16T07:36:52.359 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T07:36:52.359 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 315 k 2024-04-16T07:36:52.360 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 293 k 2024-04-16T07:36:52.361 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 99 k 2024-04-16T07:36:52.361 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-16T07:36:52.361 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-16T07:36:52.361 INFO:teuthology.orchestra.run.smithi031.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T07:36:52.361 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T07:36:52.361 INFO:teuthology.orchestra.run.smithi031.stdout:Installing weak dependencies: 2024-04-16T07:36:52.362 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-16T07:36:52.362 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:36:52.362 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-16T07:36:52.362 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:36:52.363 INFO:teuthology.orchestra.run.smithi031.stdout:Install 30 Packages 2024-04-16T07:36:52.363 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 2 Packages 2024-04-16T07:36:52.363 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:36:52.364 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 59 M 2024-04-16T07:36:52.364 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-16T07:36:52.594 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-16T07:36:52.595 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-16T07:36:54.142 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-16T07:36:54.307 INFO:teuthology.orchestra.run.smithi031.stdout:(1/32): ceph-base-19.0.0-3109.g30b0aa00.el9.x86 4.0 MB/s | 5.9 MB 00:01 2024-04-16T07:36:54.399 INFO:teuthology.orchestra.run.smithi031.stdout:(2/32): ceph-selinux-19.0.0-3109.g30b0aa00.el9. 274 kB/s | 25 kB 00:00 2024-04-16T07:36:54.590 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-16T07:36:54.600 INFO:teuthology.orchestra.run.smithi031.stdout:(3/32): libcephfs2-19.0.0-3109.g30b0aa00.el9.x8 3.9 MB/s | 797 kB 00:00 2024-04-16T07:36:54.666 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:36:54.757 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:36:54.848 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/34 2024-04-16T07:36:54.876 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:36:54.900 INFO:teuthology.orchestra.run.smithi031.stdout:(4/32): ceph-common-19.0.0-3109.g30b0aa00.el9.x 9.4 MB/s | 19 MB 00:02 2024-04-16T07:36:54.925 INFO:teuthology.orchestra.run.smithi031.stdout:(5/32): libradosstriper1-19.0.0-3109.g30b0aa00. 1.6 MB/s | 546 kB 00:00 2024-04-16T07:36:54.939 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:36:54.978 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-16T07:36:55.006 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-16T07:36:55.031 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-16T07:36:55.034 INFO:teuthology.orchestra.run.smithi031.stdout:(6/32): ceph-radosgw-19.0.0-3109.g30b0aa00.el9. 4.6 MB/s | 10 MB 00:02 2024-04-16T07:36:55.051 INFO:teuthology.orchestra.run.smithi031.stdout:(7/32): python3-ceph-argparse-19.0.0-3109.g30b0 360 kB/s | 45 kB 00:00 2024-04-16T07:36:55.054 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T07:36:55.069 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T07:36:55.126 INFO:teuthology.orchestra.run.smithi031.stdout:(8/32): python3-ceph-common-19.0.0-3109.g30b0aa 1.4 MB/s | 135 kB 00:00 2024-04-16T07:36:55.210 INFO:teuthology.orchestra.run.smithi031.stdout:(9/32): librgw2-19.0.0-3109.g30b0aa00.el9.x86_6 17 MB/s | 5.2 MB 00:00 2024-04-16T07:36:55.213 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T07:36:55.235 INFO:teuthology.orchestra.run.smithi031.stdout:(10/32): python3-rados-19.0.0-3109.g30b0aa00.el 2.8 MB/s | 315 kB 00:00 2024-04-16T07:36:55.245 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T07:36:55.260 INFO:teuthology.orchestra.run.smithi031.stdout:(11/32): python3-rbd-19.0.0-3109.g30b0aa00.el9. 5.7 MB/s | 293 kB 00:00 2024-04-16T07:36:55.327 INFO:teuthology.orchestra.run.smithi031.stdout:(12/32): python3-rgw-19.0.0-3109.g30b0aa00.el9. 1.1 MB/s | 99 kB 00:00 2024-04-16T07:36:55.377 INFO:teuthology.orchestra.run.smithi031.stdout:(13/32): python3-cephfs-19.0.0-3109.g30b0aa00.e 489 kB/s | 159 kB 00:00 2024-04-16T07:36:55.403 INFO:teuthology.orchestra.run.smithi031.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 744 kB/s | 106 kB 00:00 2024-04-16T07:36:55.428 INFO:teuthology.orchestra.run.smithi031.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 159 kB 00:00 2024-04-16T07:36:55.453 INFO:teuthology.orchestra.run.smithi031.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 590 kB/s | 44 kB 00:00 2024-04-16T07:36:55.486 INFO:teuthology.orchestra.run.smithi031.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-04-16T07:36:55.512 INFO:teuthology.orchestra.run.smithi031.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-04-16T07:36:55.537 INFO:teuthology.orchestra.run.smithi031.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 496 kB/s | 41 kB 00:00 2024-04-16T07:36:55.562 INFO:teuthology.orchestra.run.smithi031.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2024-04-16T07:36:55.588 INFO:teuthology.orchestra.run.smithi031.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 702 kB/s | 35 kB 00:00 2024-04-16T07:36:55.610 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-16T07:36:55.612 INFO:teuthology.orchestra.run.smithi031.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 620 kB/s | 46 kB 00:00 2024-04-16T07:36:55.640 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:36:55.695 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:36:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9. 12/34 2024-04-16T07:36:55.746 INFO:teuthology.orchestra.run.smithi031.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 195 kB/s | 26 kB 00:00 2024-04-16T07:36:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 13/34 2024-04-16T07:36:55.830 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 14/34 2024-04-16T07:36:55.855 INFO:teuthology.orchestra.run.smithi031.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 449 kB/s | 49 kB 00:00 2024-04-16T07:36:55.880 INFO:teuthology.orchestra.run.smithi031.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 973 kB/s | 309 kB 00:00 2024-04-16T07:36:55.918 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-16T07:36:55.943 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 16/34 2024-04-16T07:36:55.947 INFO:teuthology.orchestra.run.smithi031.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 733 kB/s | 67 kB 00:00 2024-04-16T07:36:56.031 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 16/34 2024-04-16T07:36:56.067 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-16T07:36:56.108 INFO:teuthology.orchestra.run.smithi031.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 838 kB 00:00 2024-04-16T07:36:56.148 INFO:teuthology.orchestra.run.smithi031.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 954 kB/s | 192 kB 00:00 2024-04-16T07:36:56.177 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-16T07:36:56.207 INFO:teuthology.orchestra.run.smithi031.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.4 MB/s | 4.4 MB 00:00 2024-04-16T07:36:56.210 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-16T07:36:56.251 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-16T07:36:56.251 INFO:teuthology.orchestra.run.smithi031.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-16T07:36:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-16T07:36:56.311 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-16T07:36:56.342 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-16T07:36:56.383 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-16T07:36:56.524 INFO:teuthology.orchestra.run.smithi031.stdout:(31/32): librbd1-19.0.0-3109.g30b0aa00.el9.x86_ 12 MB/s | 3.7 MB 00:00 2024-04-16T07:36:56.774 INFO:teuthology.orchestra.run.smithi031.stdout:(32/32): librados2-19.0.0-3109.g30b0aa00.el9.x8 5.4 MB/s | 3.4 MB 00:00 2024-04-16T07:36:56.777 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:36:56.778 INFO:teuthology.orchestra.run.smithi031.stdout:Total 13 MB/s | 59 MB 00:04 2024-04-16T07:36:56.778 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-16T07:36:56.824 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-16T07:36:56.824 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-16T07:36:56.943 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-16T07:36:56.967 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 26/34 2024-04-16T07:36:57.037 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 26/34 2024-04-16T07:36:57.105 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 27/34 2024-04-16T07:36:57.134 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x8 28/34 2024-04-16T07:36:57.325 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-16T07:36:57.325 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-16T07:36:58.822 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-16T07:36:59.326 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-16T07:36:59.367 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:36:59.443 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:36:59.514 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:36:59.545 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/34 2024-04-16T07:36:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:36:59.577 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:36:59.623 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:36:59.668 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-16T07:36:59.699 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-16T07:36:59.733 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-16T07:36:59.755 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T07:36:59.771 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T07:36:59.924 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T07:36:59.955 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T07:37:00.253 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:37:00.301 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 30/34 2024-04-16T07:37:00.325 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-16T07:37:00.351 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:37:00.390 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 30/34 2024-04-16T07:37:00.390 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T07:37:00.390 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T07:37:00.390 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:37:00.401 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:37:00.426 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:37:00.453 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9. 12/34 2024-04-16T07:37:00.508 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 13/34 2024-04-16T07:37:00.540 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 14/34 2024-04-16T07:37:00.630 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-16T07:37:00.654 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 16/34 2024-04-16T07:37:00.742 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 16/34 2024-04-16T07:37:00.778 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-16T07:37:00.889 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-16T07:37:00.921 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-16T07:37:00.961 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-16T07:37:00.994 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-16T07:37:01.023 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-16T07:37:01.054 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-16T07:37:01.093 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-16T07:37:01.685 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-16T07:37:01.711 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 26/34 2024-04-16T07:37:01.773 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 26/34 2024-04-16T07:37:01.845 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 27/34 2024-04-16T07:37:01.871 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x8 28/34 2024-04-16T07:37:04.655 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:37:04.693 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:37:05.421 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:37:05.463 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 30/34 2024-04-16T07:37:05.553 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 30/34 2024-04-16T07:37:05.553 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T07:37:05.553 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T07:37:05.553 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:37:05.590 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:37:10.499 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:37:10.499 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /sys 2024-04-16T07:37:10.499 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /proc 2024-04-16T07:37:10.499 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /mnt 2024-04-16T07:37:10.500 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /var/tmp 2024-04-16T07:37:10.500 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /home 2024-04-16T07:37:10.500 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /root 2024-04-16T07:37:10.500 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /tmp 2024-04-16T07:37:10.500 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:37:11.575 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/34 2024-04-16T07:37:11.608 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/34 2024-04-16T07:37:11.608 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:37:11.608 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T07:37:11.608 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:37:11.609 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:37:11.609 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:37:11.609 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T07:37:11.678 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T07:37:11.678 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:37:12.917 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:37:12.917 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/34 2024-04-16T07:37:12.917 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:37:12.917 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/34 2024-04-16T07:37:12.917 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:37:12.918 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 5/34 2024-04-16T07:37:12.918 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 6/34 2024-04-16T07:37:12.918 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 7/34 2024-04-16T07:37:12.918 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9. 8/34 2024-04-16T07:37:12.918 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x8 9/34 2024-04-16T07:37:12.918 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/34 2024-04-16T07:37:12.918 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:37:12.918 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 12/34 2024-04-16T07:37:12.918 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 13/34 2024-04-16T07:37:12.918 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-16T07:37:12.918 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-16T07:37:12.918 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-16T07:37:12.919 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-16T07:37:12.919 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-16T07:37:12.919 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-16T07:37:12.920 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-16T07:37:12.920 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-16T07:37:12.922 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-16T07:37:12.922 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-16T07:37:12.922 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-16T07:37:12.922 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-16T07:37:12.922 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-16T07:37:12.922 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-16T07:37:12.922 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-16T07:37:12.922 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-16T07:37:12.922 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-16T07:37:12.922 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:37:12.923 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-16T07:37:12.923 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/34 2024-04-16T07:37:14.031 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:37:14.031 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:37:14.031 INFO:teuthology.orchestra.run.smithi094.stdout:Upgraded: 2024-04-16T07:37:14.031 INFO:teuthology.orchestra.run.smithi094.stdout: librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:14.032 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:14.032 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-16T07:37:14.032 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T07:37:14.032 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:14.032 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:14.032 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:14.032 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:14.032 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T07:37:14.032 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T07:37:14.032 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T07:37:14.032 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:14.032 INFO:teuthology.orchestra.run.smithi094.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T07:37:14.032 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T07:37:14.032 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T07:37:14.033 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:14.033 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T07:37:14.033 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:14.033 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T07:37:14.033 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T07:37:14.033 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T07:37:14.033 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:14.033 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:14.033 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:14.033 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T07:37:14.033 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:14.033 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:14.033 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:14.033 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-16T07:37:14.033 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-16T07:37:14.034 INFO:teuthology.orchestra.run.smithi094.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-16T07:37:14.034 INFO:teuthology.orchestra.run.smithi094.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T07:37:14.034 INFO:teuthology.orchestra.run.smithi094.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T07:37:14.034 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:37:14.034 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:37:14.404 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-test 2024-04-16T07:37:14.988 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:31 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:37:15.140 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:37:15.141 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:37:15.142 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-04-16T07:37:15.142 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:37:15.142 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-16T07:37:15.142 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 84 M 2024-04-16T07:37:15.142 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-04-16T07:37:15.142 INFO:teuthology.orchestra.run.smithi094.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T07:37:15.142 INFO:teuthology.orchestra.run.smithi094.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-16T07:37:15.142 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 173 k 2024-04-16T07:37:15.142 INFO:teuthology.orchestra.run.smithi094.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-16T07:37:15.142 INFO:teuthology.orchestra.run.smithi094.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-04-16T07:37:15.142 INFO:teuthology.orchestra.run.smithi094.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T07:37:15.142 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-16T07:37:15.142 INFO:teuthology.orchestra.run.smithi094.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T07:37:15.143 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T07:37:15.143 INFO:teuthology.orchestra.run.smithi094.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-16T07:37:15.143 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:37:15.143 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-16T07:37:15.143 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:37:15.143 INFO:teuthology.orchestra.run.smithi094.stdout:Install 11 Packages 2024-04-16T07:37:15.143 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:37:15.143 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 87 M 2024-04-16T07:37:15.144 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 814 M 2024-04-16T07:37:15.144 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-16T07:37:15.678 INFO:teuthology.orchestra.run.smithi094.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-04-16T07:37:15.973 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:37:15.973 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2024-04-16T07:37:15.973 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2024-04-16T07:37:15.973 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2024-04-16T07:37:15.974 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2024-04-16T07:37:15.974 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2024-04-16T07:37:15.974 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /root 2024-04-16T07:37:15.974 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2024-04-16T07:37:15.974 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:37:15.986 INFO:teuthology.orchestra.run.smithi094.stdout:(2/11): libcephsqlite-19.0.0-3109.g30b0aa00.el9 340 kB/s | 173 kB 00:00 2024-04-16T07:37:16.020 INFO:teuthology.orchestra.run.smithi094.stdout:(3/11): protobuf-3.14.0-13.el9.x86_64.rpm 3.0 MB/s | 1.0 MB 00:00 2024-04-16T07:37:16.112 INFO:teuthology.orchestra.run.smithi094.stdout:(4/11): socat-1.7.4.1-5.el9.x86_64.rpm 2.3 MB/s | 300 kB 00:00 2024-04-16T07:37:16.129 INFO:teuthology.orchestra.run.smithi094.stdout:(5/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 577 kB/s | 63 kB 00:00 2024-04-16T07:37:16.171 INFO:teuthology.orchestra.run.smithi094.stdout:(6/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 1.5 MB/s | 90 kB 00:00 2024-04-16T07:37:16.271 INFO:teuthology.orchestra.run.smithi094.stdout:(7/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 4.6 MB/s | 469 kB 00:00 2024-04-16T07:37:16.347 INFO:teuthology.orchestra.run.smithi094.stdout:(8/11): libubsan-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-04-16T07:37:16.705 INFO:teuthology.orchestra.run.smithi094.stdout:(9/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 340 kB/s | 122 kB 00:00 2024-04-16T07:37:16.730 INFO:teuthology.orchestra.run.smithi094.stdout:(10/11): jq-1.6-16.el9.x86_64.rpm 311 kB/s | 187 kB 00:00 2024-04-16T07:37:17.072 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/34 2024-04-16T07:37:17.098 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/34 2024-04-16T07:37:17.098 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:37:17.098 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T07:37:17.098 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:37:17.099 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:37:17.099 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:37:17.100 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T07:37:17.183 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T07:37:17.183 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:37:18.412 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:37:18.412 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/34 2024-04-16T07:37:18.412 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:37:18.412 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/34 2024-04-16T07:37:18.413 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:37:18.413 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 5/34 2024-04-16T07:37:18.413 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 6/34 2024-04-16T07:37:18.413 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 7/34 2024-04-16T07:37:18.413 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9. 8/34 2024-04-16T07:37:18.413 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x8 9/34 2024-04-16T07:37:18.413 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/34 2024-04-16T07:37:18.413 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:37:18.413 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 12/34 2024-04-16T07:37:18.413 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 13/34 2024-04-16T07:37:18.413 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-16T07:37:18.413 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-16T07:37:18.413 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-16T07:37:18.413 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-16T07:37:18.414 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-16T07:37:18.414 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-16T07:37:18.414 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-16T07:37:18.414 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-16T07:37:18.414 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-16T07:37:18.414 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-16T07:37:18.414 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-16T07:37:18.414 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-16T07:37:18.414 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-16T07:37:18.414 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-16T07:37:18.414 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-16T07:37:18.414 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-16T07:37:18.414 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-16T07:37:18.414 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:37:18.415 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-16T07:37:18.415 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/34 2024-04-16T07:37:19.448 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:37:19.449 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:37:19.449 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2024-04-16T07:37:19.449 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:19.449 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:19.450 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-16T07:37:19.450 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T07:37:19.450 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:19.450 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:19.450 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:19.450 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:19.450 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T07:37:19.450 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T07:37:19.450 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T07:37:19.450 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:19.450 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T07:37:19.450 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T07:37:19.450 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T07:37:19.450 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:19.451 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T07:37:19.451 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:19.451 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T07:37:19.451 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T07:37:19.451 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T07:37:19.451 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:19.451 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:19.451 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:19.451 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T07:37:19.451 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:19.451 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:19.451 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:37:19.451 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-16T07:37:19.451 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-16T07:37:19.452 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-16T07:37:19.452 INFO:teuthology.orchestra.run.smithi031.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T07:37:19.452 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T07:37:19.452 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:37:19.452 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:37:19.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-test 2024-04-16T07:37:20.387 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:31 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:37:20.547 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:37:20.548 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:37:20.548 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-16T07:37:20.548 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:37:20.549 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-16T07:37:20.549 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 84 M 2024-04-16T07:37:20.549 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-16T07:37:20.549 INFO:teuthology.orchestra.run.smithi031.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T07:37:20.549 INFO:teuthology.orchestra.run.smithi031.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-16T07:37:20.549 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 173 k 2024-04-16T07:37:20.549 INFO:teuthology.orchestra.run.smithi031.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-16T07:37:20.549 INFO:teuthology.orchestra.run.smithi031.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-04-16T07:37:20.549 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T07:37:20.549 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-16T07:37:20.549 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T07:37:20.549 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T07:37:20.550 INFO:teuthology.orchestra.run.smithi031.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-16T07:37:20.550 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:37:20.550 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-16T07:37:20.550 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:37:20.550 INFO:teuthology.orchestra.run.smithi031.stdout:Install 11 Packages 2024-04-16T07:37:20.550 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:37:20.550 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 87 M 2024-04-16T07:37:20.551 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 814 M 2024-04-16T07:37:20.551 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-16T07:37:21.359 INFO:teuthology.orchestra.run.smithi031.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 986 kB/s | 217 kB 00:00 2024-04-16T07:37:21.443 INFO:teuthology.orchestra.run.smithi031.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-04-16T07:37:21.485 INFO:teuthology.orchestra.run.smithi031.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-04-16T07:37:21.518 INFO:teuthology.orchestra.run.smithi031.stdout:(4/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-04-16T07:37:21.560 INFO:teuthology.orchestra.run.smithi031.stdout:(5/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 2.1 MB/s | 90 kB 00:00 2024-04-16T07:37:21.586 INFO:teuthology.orchestra.run.smithi031.stdout:(6/11): libcephsqlite-19.0.0-3109.g30b0aa00.el9 388 kB/s | 173 kB 00:00 2024-04-16T07:37:21.794 INFO:teuthology.orchestra.run.smithi031.stdout:(7/11): jq-1.6-16.el9.x86_64.rpm 801 kB/s | 187 kB 00:00 2024-04-16T07:37:22.278 INFO:teuthology.orchestra.run.smithi031.stdout:(8/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 677 kB/s | 469 kB 00:00 2024-04-16T07:37:22.303 INFO:teuthology.orchestra.run.smithi031.stdout:(9/11): libubsan-11.4.1-3.el9.x86_64.rpm 367 kB/s | 187 kB 00:00 2024-04-16T07:37:22.662 INFO:teuthology.orchestra.run.smithi031.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 318 kB/s | 122 kB 00:00 2024-04-16T07:37:47.902 INFO:teuthology.orchestra.run.smithi031.stdout:(11/11): ceph-test-19.0.0-3109.g30b0aa00.el9.x8 3.1 MB/s | 84 MB 00:26 2024-04-16T07:37:47.904 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:37:47.904 INFO:teuthology.orchestra.run.smithi031.stdout:Total 3.2 MB/s | 87 MB 00:27 2024-04-16T07:37:47.941 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-16T07:37:47.964 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-16T07:37:47.964 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-16T07:37:48.610 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-16T07:37:48.610 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-16T07:37:49.667 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-16T07:37:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-04-16T07:37:49.760 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-04-16T07:37:49.785 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T07:37:49.857 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T07:37:49.905 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-04-16T07:37:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-04-16T07:37:49.991 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-04-16T07:37:50.022 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-04-16T07:37:50.058 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-04-16T07:37:50.098 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-04-16T07:37:50.232 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/11 2024-04-16T07:38:02.179 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/11 2024-04-16T07:38:02.260 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/11 2024-04-16T07:38:08.351 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/11 2024-04-16T07:38:08.351 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/11 2024-04-16T07:38:08.351 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/11 2024-04-16T07:38:08.351 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-04-16T07:38:08.351 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-04-16T07:38:08.352 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-04-16T07:38:08.352 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-04-16T07:38:08.352 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-04-16T07:38:08.352 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-04-16T07:38:08.352 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-04-16T07:38:08.352 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-04-16T07:38:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-04-16T07:38:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:38:08.713 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-16T07:38:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: jq-1.6-16.el9.x86_64 2024-04-16T07:38:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-16T07:38:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-16T07:38:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-16T07:38:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T07:38:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-16T07:38:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T07:38:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T07:38:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-16T07:38:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:38:08.714 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:38:09.006 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph 2024-04-16T07:38:09.596 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:20 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:38:09.757 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:38:09.760 INFO:teuthology.orchestra.run.smithi031.stdout:========================================================================================= 2024-04-16T07:38:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-16T07:38:09.760 INFO:teuthology.orchestra.run.smithi031.stdout:========================================================================================= 2024-04-16T07:38:09.760 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-16T07:38:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 6.5 k 2024-04-16T07:38:09.760 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-16T07:38:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 2.2 M 2024-04-16T07:38:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 1.6 M 2024-04-16T07:38:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 249 k 2024-04-16T07:38:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.0 M 2024-04-16T07:38:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 71 M 2024-04-16T07:38:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-16T07:38:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-16T07:38:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-16T07:38:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T07:38:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T07:38:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T07:38:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T07:38:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T07:38:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T07:38:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T07:38:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T07:38:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T07:38:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T07:38:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T07:38:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T07:38:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T07:38:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T07:38:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T07:38:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T07:38:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T07:38:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T07:38:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T07:38:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T07:38:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T07:38:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T07:38:09.763 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T07:38:09.763 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T07:38:09.763 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T07:38:09.763 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T07:38:09.763 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T07:38:09.763 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:38:09.763 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-16T07:38:09.763 INFO:teuthology.orchestra.run.smithi031.stdout:========================================================================================= 2024-04-16T07:38:09.763 INFO:teuthology.orchestra.run.smithi031.stdout:Install 36 Packages 2024-04-16T07:38:09.763 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:38:09.764 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 84 M 2024-04-16T07:38:09.764 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 922 M 2024-04-16T07:38:09.764 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-16T07:38:10.781 INFO:teuthology.orchestra.run.smithi031.stdout:(1/36): ceph-19.0.0-3109.g30b0aa00.el9.x86_64.r 17 kB/s | 6.5 kB 00:00 2024-04-16T07:38:13.231 INFO:teuthology.orchestra.run.smithi031.stdout:(2/36): ceph-mds-19.0.0-3109.g30b0aa00.el9.x86_ 788 kB/s | 2.2 MB 00:02 2024-04-16T07:38:13.893 INFO:teuthology.orchestra.run.smithi031.stdout:(3/36): ceph-mgr-19.0.0-3109.g30b0aa00.el9.x86_ 455 kB/s | 1.6 MB 00:03 2024-04-16T07:38:14.439 INFO:teuthology.orchestra.run.smithi031.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3109.g30b0 455 kB/s | 249 kB 00:00 2024-04-16T07:38:14.631 INFO:teuthology.orchestra.run.smithi031.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 976 kB/s | 187 kB 00:00 2024-04-16T07:38:14.698 INFO:teuthology.orchestra.run.smithi031.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-04-16T07:38:14.748 INFO:teuthology.orchestra.run.smithi031.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 634 kB/s | 32 kB 00:00 2024-04-16T07:38:14.790 INFO:teuthology.orchestra.run.smithi031.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-04-16T07:38:14.825 INFO:teuthology.orchestra.run.smithi031.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-04-16T07:38:14.882 INFO:teuthology.orchestra.run.smithi031.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-04-16T07:38:14.949 INFO:teuthology.orchestra.run.smithi031.stdout:(11/36): ceph-mon-19.0.0-3109.g30b0aa00.el9.x86 1.2 MB/s | 5.0 MB 00:04 2024-04-16T07:38:15.008 INFO:teuthology.orchestra.run.smithi031.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 9.1 MB/s | 1.1 MB 00:00 2024-04-16T07:38:15.034 INFO:teuthology.orchestra.run.smithi031.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-04-16T07:38:15.058 INFO:teuthology.orchestra.run.smithi031.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-04-16T07:38:15.084 INFO:teuthology.orchestra.run.smithi031.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-04-16T07:38:15.234 INFO:teuthology.orchestra.run.smithi031.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 149 kB/s | 22 kB 00:00 2024-04-16T07:38:15.334 INFO:teuthology.orchestra.run.smithi031.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 454 kB/s | 125 kB 00:00 2024-04-16T07:38:15.468 INFO:teuthology.orchestra.run.smithi031.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 647 kB/s | 151 kB 00:00 2024-04-16T07:38:15.493 INFO:teuthology.orchestra.run.smithi031.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 276 kB/s | 43 kB 00:00 2024-04-16T07:38:15.544 INFO:teuthology.orchestra.run.smithi031.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-16T07:38:15.569 INFO:teuthology.orchestra.run.smithi031.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.7 MB/s | 358 kB 00:00 2024-04-16T07:38:15.619 INFO:teuthology.orchestra.run.smithi031.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 144 kB/s | 11 kB 00:00 2024-04-16T07:38:15.636 INFO:teuthology.orchestra.run.smithi031.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 264 kB/s | 18 kB 00:00 2024-04-16T07:38:15.662 INFO:teuthology.orchestra.run.smithi031.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 550 kB/s | 23 kB 00:00 2024-04-16T07:38:15.678 INFO:teuthology.orchestra.run.smithi031.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 461 kB/s | 19 kB 00:00 2024-04-16T07:38:15.704 INFO:teuthology.orchestra.run.smithi031.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 471 kB/s | 20 kB 00:00 2024-04-16T07:38:15.737 INFO:teuthology.orchestra.run.smithi031.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 790 kB/s | 46 kB 00:00 2024-04-16T07:38:15.771 INFO:teuthology.orchestra.run.smithi031.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-04-16T07:38:15.813 INFO:teuthology.orchestra.run.smithi031.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-16T07:38:15.830 INFO:teuthology.orchestra.run.smithi031.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-04-16T07:38:15.880 INFO:teuthology.orchestra.run.smithi031.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-04-16T07:38:15.905 INFO:teuthology.orchestra.run.smithi031.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 475 kB/s | 36 kB 00:00 2024-04-16T07:38:15.947 INFO:teuthology.orchestra.run.smithi031.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-04-16T07:38:15.972 INFO:teuthology.orchestra.run.smithi031.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.3 MB/s | 427 kB 00:00 2024-04-16T07:38:16.023 INFO:teuthology.orchestra.run.smithi031.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 269 kB/s | 20 kB 00:00 2024-04-16T07:38:54.091 INFO:teuthology.orchestra.run.smithi094.stdout:(11/11): ceph-test-19.0.0-3109.g30b0aa00.el9.x8 875 kB/s | 84 MB 01:38 2024-04-16T07:38:54.092 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:38:54.093 INFO:teuthology.orchestra.run.smithi094.stdout:Total 901 kB/s | 87 MB 01:38 2024-04-16T07:38:54.127 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-16T07:38:54.150 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-16T07:38:54.150 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-16T07:38:54.781 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-16T07:38:54.782 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-16T07:38:55.807 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-16T07:38:55.855 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-04-16T07:38:55.890 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-04-16T07:38:55.916 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T07:38:55.996 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T07:38:56.037 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-04-16T07:38:56.084 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-04-16T07:38:56.129 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-04-16T07:38:56.161 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-04-16T07:38:56.200 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-04-16T07:38:56.237 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-04-16T07:38:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/11 2024-04-16T07:39:07.789 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/11 2024-04-16T07:39:07.872 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/11 2024-04-16T07:39:14.343 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/11 2024-04-16T07:39:14.344 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/11 2024-04-16T07:39:14.344 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/11 2024-04-16T07:39:14.344 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-04-16T07:39:14.344 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-04-16T07:39:14.344 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-04-16T07:39:14.344 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-04-16T07:39:14.344 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-04-16T07:39:14.344 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-04-16T07:39:14.344 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-04-16T07:39:14.344 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-04-16T07:39:14.717 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-04-16T07:39:14.717 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:39:14.717 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-16T07:39:14.717 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:39:14.717 INFO:teuthology.orchestra.run.smithi094.stdout: jq-1.6-16.el9.x86_64 2024-04-16T07:39:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-16T07:39:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:39:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-16T07:39:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-16T07:39:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T07:39:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-16T07:39:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T07:39:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T07:39:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-16T07:39:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:39:14.718 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:39:15.023 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph 2024-04-16T07:39:15.609 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:32 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:39:15.762 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:39:15.765 INFO:teuthology.orchestra.run.smithi094.stdout:========================================================================================= 2024-04-16T07:39:15.765 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-04-16T07:39:15.765 INFO:teuthology.orchestra.run.smithi094.stdout:========================================================================================= 2024-04-16T07:39:15.765 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-16T07:39:15.765 INFO:teuthology.orchestra.run.smithi094.stdout: ceph x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 6.5 k 2024-04-16T07:39:15.765 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-04-16T07:39:15.765 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 2.2 M 2024-04-16T07:39:15.765 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 1.6 M 2024-04-16T07:39:15.765 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 249 k 2024-04-16T07:39:15.765 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.0 M 2024-04-16T07:39:15.765 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 71 M 2024-04-16T07:39:15.765 INFO:teuthology.orchestra.run.smithi094.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-16T07:39:15.766 INFO:teuthology.orchestra.run.smithi094.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-16T07:39:15.766 INFO:teuthology.orchestra.run.smithi094.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-16T07:39:15.766 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T07:39:15.766 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T07:39:15.766 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T07:39:15.766 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T07:39:15.766 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T07:39:15.766 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T07:39:15.766 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T07:39:15.766 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T07:39:15.766 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T07:39:15.766 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T07:39:15.767 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T07:39:15.767 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T07:39:15.767 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T07:39:15.767 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T07:39:15.767 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T07:39:15.767 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T07:39:15.767 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T07:39:15.767 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T07:39:15.767 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T07:39:15.767 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T07:39:15.767 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T07:39:15.767 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T07:39:15.767 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T07:39:15.768 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T07:39:15.768 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T07:39:15.768 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T07:39:15.768 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T07:39:15.768 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:39:15.768 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-16T07:39:15.768 INFO:teuthology.orchestra.run.smithi094.stdout:========================================================================================= 2024-04-16T07:39:15.768 INFO:teuthology.orchestra.run.smithi094.stdout:Install 36 Packages 2024-04-16T07:39:15.768 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:39:15.769 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 84 M 2024-04-16T07:39:15.769 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 922 M 2024-04-16T07:39:15.769 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-16T07:39:16.797 INFO:teuthology.orchestra.run.smithi094.stdout:(1/36): ceph-19.0.0-3109.g30b0aa00.el9.x86_64.r 18 kB/s | 6.5 kB 00:00 2024-04-16T07:39:18.314 INFO:teuthology.orchestra.run.smithi094.stdout:(2/36): ceph-mds-19.0.0-3109.g30b0aa00.el9.x86_ 1.2 MB/s | 2.2 MB 00:01 2024-04-16T07:39:18.423 INFO:teuthology.orchestra.run.smithi094.stdout:(3/36): ceph-mgr-19.0.0-3109.g30b0aa00.el9.x86_ 802 kB/s | 1.6 MB 00:01 2024-04-16T07:39:18.765 INFO:teuthology.orchestra.run.smithi094.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3109.g30b0 728 kB/s | 249 kB 00:00 2024-04-16T07:39:18.881 INFO:teuthology.orchestra.run.smithi094.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-04-16T07:39:18.923 INFO:teuthology.orchestra.run.smithi094.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-04-16T07:39:18.948 INFO:teuthology.orchestra.run.smithi094.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-04-16T07:39:18.974 INFO:teuthology.orchestra.run.smithi094.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-04-16T07:39:18.999 INFO:teuthology.orchestra.run.smithi094.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-04-16T07:39:19.041 INFO:teuthology.orchestra.run.smithi094.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-04-16T07:39:19.116 INFO:teuthology.orchestra.run.smithi094.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-04-16T07:39:19.142 INFO:teuthology.orchestra.run.smithi094.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 4.0 MB/s | 103 kB 00:00 2024-04-16T07:39:19.176 INFO:teuthology.orchestra.run.smithi094.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-04-16T07:39:19.251 INFO:teuthology.orchestra.run.smithi094.stdout:(14/36): ceph-mon-19.0.0-3109.g30b0aa00.el9.x86 2.0 MB/s | 5.0 MB 00:02 2024-04-16T07:39:19.310 INFO:teuthology.orchestra.run.smithi094.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.4 MB/s | 187 kB 00:00 2024-04-16T07:39:19.501 INFO:teuthology.orchestra.run.smithi094.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 500 kB/s | 125 kB 00:00 2024-04-16T07:39:19.527 INFO:teuthology.orchestra.run.smithi094.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 103 kB/s | 22 kB 00:00 2024-04-16T07:39:19.835 INFO:teuthology.orchestra.run.smithi094.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 141 kB/s | 43 kB 00:00 2024-04-16T07:39:19.902 INFO:teuthology.orchestra.run.smithi094.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 377 kB/s | 151 kB 00:00 2024-04-16T07:39:20.046 INFO:teuthology.orchestra.run.smithi094.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 823 kB/s | 172 kB 00:00 2024-04-16T07:39:20.072 INFO:teuthology.orchestra.run.smithi094.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-04-16T07:39:20.112 INFO:teuthology.orchestra.run.smithi094.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 159 kB/s | 11 kB 00:00 2024-04-16T07:39:20.137 INFO:teuthology.orchestra.run.smithi094.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 264 kB/s | 18 kB 00:00 2024-04-16T07:39:20.182 INFO:teuthology.orchestra.run.smithi094.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2024-04-16T07:39:20.204 INFO:teuthology.orchestra.run.smithi094.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 290 kB/s | 19 kB 00:00 2024-04-16T07:39:20.246 INFO:teuthology.orchestra.run.smithi094.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 293 kB/s | 20 kB 00:00 2024-04-16T07:39:20.272 INFO:teuthology.orchestra.run.smithi094.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 693 kB/s | 46 kB 00:00 2024-04-16T07:39:20.330 INFO:teuthology.orchestra.run.smithi094.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 941 kB/s | 79 kB 00:00 2024-04-16T07:39:20.372 INFO:teuthology.orchestra.run.smithi094.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-04-16T07:39:20.398 INFO:teuthology.orchestra.run.smithi094.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 246 kB/s | 16 kB 00:00 2024-04-16T07:39:20.449 INFO:teuthology.orchestra.run.smithi094.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-04-16T07:39:20.465 INFO:teuthology.orchestra.run.smithi094.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 535 kB/s | 36 kB 00:00 2024-04-16T07:39:20.540 INFO:teuthology.orchestra.run.smithi094.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-16T07:39:20.615 INFO:teuthology.orchestra.run.smithi094.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-04-16T07:39:20.649 INFO:teuthology.orchestra.run.smithi094.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.3 MB/s | 427 kB 00:00 2024-04-16T07:39:56.684 INFO:teuthology.orchestra.run.smithi031.stdout:(36/36): ceph-osd-19.0.0-3109.g30b0aa00.el9.x86 702 kB/s | 71 MB 01:43 2024-04-16T07:39:56.688 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:39:56.688 INFO:teuthology.orchestra.run.smithi031.stdout:Total 807 kB/s | 84 MB 01:46 2024-04-16T07:39:56.707 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-16T07:39:56.749 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-16T07:39:56.750 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-16T07:39:57.516 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-16T07:39:57.516 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-16T07:39:59.262 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-16T07:39:59.327 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-16T07:39:59.379 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-16T07:39:59.405 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-16T07:39:59.468 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-16T07:39:59.504 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-16T07:39:59.537 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-16T07:39:59.631 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-16T07:39:59.708 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-16T07:39:59.756 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-16T07:39:59.829 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-16T07:39:59.882 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-16T07:39:59.920 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-16T07:39:59.969 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-16T07:40:00.025 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-16T07:40:00.224 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-16T07:40:00.273 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-16T07:40:00.335 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-16T07:40:00.371 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-16T07:40:00.401 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-16T07:40:00.480 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-16T07:40:00.512 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T07:40:00.545 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-16T07:40:00.586 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-16T07:40:00.612 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-16T07:40:00.638 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-16T07:40:00.730 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-16T07:40:00.769 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-16T07:40:00.843 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-16T07:40:00.983 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-16T07:40:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T07:40:01.305 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9. 31/36 2024-04-16T07:40:01.331 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/36 2024-04-16T07:40:01.367 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/36 2024-04-16T07:40:01.367 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:40:01.367 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T07:40:01.367 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:40:01.368 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:40:01.368 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:08.145 INFO:teuthology.orchestra.run.smithi094.stdout:(36/36): ceph-osd-19.0.0-3109.g30b0aa00.el9.x86 1.4 MB/s | 71 MB 00:49 2024-04-16T07:40:08.150 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:40:08.150 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.6 MB/s | 84 MB 00:52 2024-04-16T07:40:08.173 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-16T07:40:08.213 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-16T07:40:08.213 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-16T07:40:08.965 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-16T07:40:08.966 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-16T07:40:10.585 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-16T07:40:10.647 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-16T07:40:10.689 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-16T07:40:10.725 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-16T07:40:10.795 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-16T07:40:10.849 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-16T07:40:10.882 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-16T07:40:10.974 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-16T07:40:11.051 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-16T07:40:11.093 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-16T07:40:11.164 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-16T07:40:11.218 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-16T07:40:11.257 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-16T07:40:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-16T07:40:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-16T07:40:11.551 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-16T07:40:11.612 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-16T07:40:11.651 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-16T07:40:11.685 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-16T07:40:11.715 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-16T07:40:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-16T07:40:11.826 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T07:40:11.859 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-16T07:40:11.901 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-16T07:40:11.934 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-16T07:40:11.969 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-16T07:40:12.057 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-16T07:40:12.092 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-16T07:40:12.162 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-16T07:40:12.298 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-16T07:40:12.424 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T07:40:12.611 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9. 31/36 2024-04-16T07:40:12.639 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/36 2024-04-16T07:40:12.665 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/36 2024-04-16T07:40:12.665 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:40:12.665 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T07:40:12.665 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:40:12.665 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:40:12.665 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:12.840 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/36 2024-04-16T07:40:12.871 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/36 2024-04-16T07:40:12.872 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:40:12.872 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T07:40:12.872 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:40:12.872 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:40:12.873 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:13.432 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 34/36 2024-04-16T07:40:13.454 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 34/36 2024-04-16T07:40:13.454 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:40:13.454 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T07:40:13.454 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:40:13.455 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:40:13.455 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:13.699 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 35/36 2024-04-16T07:40:13.722 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 35/36 2024-04-16T07:40:13.722 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:40:13.722 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T07:40:13.722 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:40:13.723 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:40:13.723 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:13.790 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 36/36 2024-04-16T07:40:20.235 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 36/36 2024-04-16T07:40:20.235 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/36 2024-04-16T07:40:20.235 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/36 2024-04-16T07:40:20.235 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/36 2024-04-16T07:40:20.235 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/36 2024-04-16T07:40:20.235 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 5/36 2024-04-16T07:40:20.235 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9. 6/36 2024-04-16T07:40:20.235 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-16T07:40:20.235 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-16T07:40:20.236 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-16T07:40:20.236 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-16T07:40:20.236 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-16T07:40:20.236 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-16T07:40:20.236 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-16T07:40:20.236 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-16T07:40:20.236 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-16T07:40:20.236 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-16T07:40:20.236 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-16T07:40:20.236 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-16T07:40:20.236 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-16T07:40:20.236 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-16T07:40:20.236 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T07:40:20.236 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-16T07:40:20.237 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-16T07:40:20.237 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-16T07:40:20.237 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-16T07:40:20.238 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-16T07:40:20.238 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-16T07:40:20.238 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-16T07:40:20.238 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-16T07:40:20.238 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T07:40:20.238 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-16T07:40:20.238 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-16T07:40:20.238 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-16T07:40:20.238 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-16T07:40:20.239 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-16T07:40:21.492 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-16T07:40:21.492 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:21.492 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-16T07:40:21.492 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:40:21.492 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:40:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:40:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:40:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:40:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:40:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: lua-5.4.4-4.el9.x86_64 2024-04-16T07:40:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-16T07:40:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-16T07:40:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T07:40:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T07:40:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T07:40:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T07:40:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T07:40:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T07:40:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T07:40:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T07:40:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T07:40:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T07:40:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T07:40:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T07:40:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T07:40:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T07:40:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T07:40:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T07:40:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T07:40:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T07:40:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T07:40:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T07:40:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T07:40:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T07:40:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T07:40:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T07:40:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T07:40:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T07:40:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T07:40:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:21.495 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:40:21.792 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-base 2024-04-16T07:40:22.386 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:33 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:40:22.487 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:40:22.542 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:40:22.544 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-16T07:40:22.544 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:40:22.618 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install cephadm 2024-04-16T07:40:23.203 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:34 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:40:23.361 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:40:23.361 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:40:23.361 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-16T07:40:23.361 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:40:23.361 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-16T07:40:23.361 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 762 k 2024-04-16T07:40:23.361 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:23.361 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-16T07:40:23.361 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:40:23.361 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-04-16T07:40:23.361 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:23.362 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 762 k 2024-04-16T07:40:23.362 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 769 k 2024-04-16T07:40:23.362 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-16T07:40:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/36 2024-04-16T07:40:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/36 2024-04-16T07:40:23.433 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:40:23.433 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T07:40:23.433 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:40:23.433 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:40:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:23.973 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 34/36 2024-04-16T07:40:23.997 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 34/36 2024-04-16T07:40:23.997 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:40:23.997 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T07:40:23.997 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:40:23.997 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:40:23.997 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:24.232 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 35/36 2024-04-16T07:40:24.256 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 35/36 2024-04-16T07:40:24.256 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:40:24.256 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T07:40:24.256 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:40:24.256 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:40:24.256 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:24.321 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 36/36 2024-04-16T07:40:25.062 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm-19.0.0-3109.g30b0aa00.el9.noarch.rpm 449 kB/s | 762 kB 00:01 2024-04-16T07:40:25.062 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:40:25.062 INFO:teuthology.orchestra.run.smithi031.stdout:Total 448 kB/s | 762 kB 00:01 2024-04-16T07:40:25.063 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-16T07:40:25.070 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-16T07:40:25.070 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-16T07:40:25.076 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-16T07:40:25.077 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-16T07:40:25.158 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-16T07:40:25.627 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:40:25.720 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:40:26.118 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:40:26.398 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:40:26.398 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:26.398 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-16T07:40:26.398 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:40:26.398 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:26.398 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:40:26.601 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T07:40:27.196 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:38 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:40:27.354 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:40:27.355 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:40:27.355 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-04-16T07:40:27.355 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:40:27.355 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-16T07:40:27.355 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 157 k 2024-04-16T07:40:27.355 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:27.355 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-16T07:40:27.355 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:40:27.355 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-04-16T07:40:27.355 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:27.356 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 157 k 2024-04-16T07:40:27.356 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 640 k 2024-04-16T07:40:27.356 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-16T07:40:28.350 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-immutable-object-cache-19.0.0-3109.g30b0aa 158 kB/s | 157 kB 00:00 2024-04-16T07:40:28.355 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:40:28.356 INFO:teuthology.orchestra.run.smithi031.stdout:Total 158 kB/s | 157 kB 00:00 2024-04-16T07:40:28.356 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-16T07:40:28.364 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-16T07:40:28.364 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-16T07:40:28.423 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-16T07:40:28.423 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-16T07:40:28.578 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-16T07:40:28.677 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:40:28.702 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:40:28.702 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:40:28.702 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T07:40:28.702 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:29.682 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:40:29.683 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:29.683 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-16T07:40:29.683 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:40:29.683 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:29.683 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:40:29.906 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr 2024-04-16T07:40:30.501 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:41 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:40:30.603 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:40:30.658 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:40:30.659 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-16T07:40:30.659 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:40:30.731 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T07:40:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 36/36 2024-04-16T07:40:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/36 2024-04-16T07:40:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/36 2024-04-16T07:40:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/36 2024-04-16T07:40:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/36 2024-04-16T07:40:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 5/36 2024-04-16T07:40:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9. 6/36 2024-04-16T07:40:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-16T07:40:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-16T07:40:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-16T07:40:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-16T07:40:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-16T07:40:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-16T07:40:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-16T07:40:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-16T07:40:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-16T07:40:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-16T07:40:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-16T07:40:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-16T07:40:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-16T07:40:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-16T07:40:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T07:40:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-16T07:40:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-16T07:40:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-16T07:40:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-16T07:40:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-16T07:40:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-16T07:40:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-16T07:40:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-16T07:40:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T07:40:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-16T07:40:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-16T07:40:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-16T07:40:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-16T07:40:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-16T07:40:31.330 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:42 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:40:31.489 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:40:31.490 INFO:teuthology.orchestra.run.smithi031.stdout:====================================================================================== 2024-04-16T07:40:31.490 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-16T07:40:31.490 INFO:teuthology.orchestra.run.smithi031.stdout:====================================================================================== 2024-04-16T07:40:31.490 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-16T07:40:31.490 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 3.5 M 2024-04-16T07:40:31.490 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-16T07:40:31.490 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-16T07:40:31.490 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 34 k 2024-04-16T07:40:31.490 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 16 k 2024-04-16T07:40:31.490 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-16T07:40:31.491 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-16T07:40:31.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-16T07:40:31.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-16T07:40:31.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-16T07:40:31.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T07:40:31.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T07:40:31.491 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:31.491 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-16T07:40:31.491 INFO:teuthology.orchestra.run.smithi031.stdout:====================================================================================== 2024-04-16T07:40:31.491 INFO:teuthology.orchestra.run.smithi031.stdout:Install 11 Packages 2024-04-16T07:40:31.491 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:31.492 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 7.6 M 2024-04-16T07:40:31.492 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 100 M 2024-04-16T07:40:31.492 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-16T07:40:32.172 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-16T07:40:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:32.173 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-16T07:40:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:40:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:40:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:40:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:40:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:40:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:40:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: lua-5.4.4-4.el9.x86_64 2024-04-16T07:40:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-16T07:40:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-16T07:40:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T07:40:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T07:40:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T07:40:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T07:40:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T07:40:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T07:40:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T07:40:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T07:40:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T07:40:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T07:40:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T07:40:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T07:40:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T07:40:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T07:40:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T07:40:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T07:40:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T07:40:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T07:40:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T07:40:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T07:40:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T07:40:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T07:40:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T07:40:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T07:40:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T07:40:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T07:40:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T07:40:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:32.175 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:40:32.424 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-base 2024-04-16T07:40:32.429 INFO:teuthology.orchestra.run.smithi031.stdout:(1/11): ceph-grafana-dashboards-19.0.0-3109.g30 62 kB/s | 34 kB 00:00 2024-04-16T07:40:32.455 INFO:teuthology.orchestra.run.smithi031.stdout:(2/11): ceph-prometheus-alerts-19.0.0-3109.g30b 29 kB/s | 16 kB 00:00 2024-04-16T07:40:32.605 INFO:teuthology.orchestra.run.smithi031.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 238 kB 00:00 2024-04-16T07:40:32.747 INFO:teuthology.orchestra.run.smithi031.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 2.9 MB/s | 863 kB 00:00 2024-04-16T07:40:32.998 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:49 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:40:33.097 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:40:33.148 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:40:33.149 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-04-16T07:40:33.149 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:40:33.210 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install cephadm 2024-04-16T07:40:33.213 INFO:teuthology.orchestra.run.smithi031.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 42 kB/s | 19 kB 00:00 2024-04-16T07:40:33.782 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:50 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:40:33.856 INFO:teuthology.orchestra.run.smithi031.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 438 kB/s | 548 kB 00:01 2024-04-16T07:40:33.932 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:40:33.933 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:40:33.933 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-04-16T07:40:33.933 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:40:33.933 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-16T07:40:33.933 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 762 k 2024-04-16T07:40:33.933 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:33.933 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-16T07:40:33.933 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:40:33.933 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-04-16T07:40:33.933 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:33.933 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 762 k 2024-04-16T07:40:33.933 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 769 k 2024-04-16T07:40:33.933 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-16T07:40:34.122 INFO:teuthology.orchestra.run.smithi031.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 542 kB/s | 144 kB 00:00 2024-04-16T07:40:34.248 INFO:teuthology.orchestra.run.smithi031.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 246 kB/s | 31 kB 00:00 2024-04-16T07:40:34.531 INFO:teuthology.orchestra.run.smithi031.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 666 kB/s | 188 kB 00:00 2024-04-16T07:40:34.590 INFO:teuthology.orchestra.run.smithi031.stdout:(10/11): ceph-mgr-dashboard-19.0.0-3109.g30b0aa 1.3 MB/s | 3.5 MB 00:02 2024-04-16T07:40:35.778 INFO:teuthology.orchestra.run.smithi094.stdout:cephadm-19.0.0-3109.g30b0aa00.el9.noarch.rpm 413 kB/s | 762 kB 00:01 2024-04-16T07:40:35.778 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:40:35.778 INFO:teuthology.orchestra.run.smithi094.stdout:Total 413 kB/s | 762 kB 00:01 2024-04-16T07:40:35.779 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-16T07:40:35.785 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-16T07:40:35.785 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-16T07:40:35.791 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-16T07:40:35.792 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-16T07:40:35.872 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-16T07:40:36.165 INFO:teuthology.orchestra.run.smithi031.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 707 kB/s | 2.0 MB 00:02 2024-04-16T07:40:36.170 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:40:36.171 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.6 MB/s | 7.6 MB 00:04 2024-04-16T07:40:36.192 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-16T07:40:36.212 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-16T07:40:36.212 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-16T07:40:36.243 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:40:36.349 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:40:36.398 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-16T07:40:36.399 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-16T07:40:36.720 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:40:36.894 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-16T07:40:36.973 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-16T07:40:36.996 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:40:36.996 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:36.996 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-16T07:40:36.996 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:40:36.996 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:36.996 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:40:37.001 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-16T07:40:37.092 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-16T07:40:37.198 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T07:40:37.232 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-16T07:40:37.288 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-16T07:40:37.375 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-16T07:40:37.424 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-16T07:40:37.460 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-16T07:40:37.493 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9 9/11 2024-04-16T07:40:37.784 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:54 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:40:37.940 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:40:37.940 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:40:37.940 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-04-16T07:40:37.941 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:40:37.941 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-16T07:40:37.941 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 157 k 2024-04-16T07:40:37.941 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:37.941 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-16T07:40:37.941 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:40:37.941 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-04-16T07:40:37.941 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:37.941 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 157 k 2024-04-16T07:40:37.941 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 640 k 2024-04-16T07:40:37.941 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-16T07:40:38.596 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el 10/11 2024-04-16T07:40:38.634 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 11/11 2024-04-16T07:40:38.735 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-immutable-object-cache-19.0.0-3109.g30b0aa 198 kB/s | 157 kB 00:00 2024-04-16T07:40:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:40:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:Total 198 kB/s | 157 kB 00:00 2024-04-16T07:40:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-16T07:40:38.746 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-16T07:40:38.746 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-16T07:40:38.801 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-16T07:40:38.802 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-16T07:40:38.947 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-16T07:40:39.028 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:40:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:40:39.053 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:40:39.053 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T07:40:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:39.888 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 11/11 2024-04-16T07:40:39.888 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el 1/11 2024-04-16T07:40:39.888 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 2/11 2024-04-16T07:40:39.888 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9 3/11 2024-04-16T07:40:39.888 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-16T07:40:39.888 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-16T07:40:39.889 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-16T07:40:39.889 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-16T07:40:39.889 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-16T07:40:39.889 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-16T07:40:39.889 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-16T07:40:40.036 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:40:40.037 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:40.037 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-16T07:40:40.037 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:40:40.037 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:40.037 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:40:40.297 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr 2024-04-16T07:40:40.366 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-16T07:40:40.366 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:40.366 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-16T07:40:40.366 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-16T07:40:40.366 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:40:40.366 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:40:40.366 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:40:40.366 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-16T07:40:40.366 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-16T07:40:40.367 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-16T07:40:40.367 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-16T07:40:40.367 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-16T07:40:40.367 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T07:40:40.367 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T07:40:40.367 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:40.367 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:40:40.723 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T07:40:40.878 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:57 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:40:40.976 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:40:41.028 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:40:41.029 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-04-16T07:40:41.029 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:40:41.095 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T07:40:41.319 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:52 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:40:41.479 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:40:41.480 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2024-04-16T07:40:41.480 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-16T07:40:41.481 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2024-04-16T07:40:41.481 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-16T07:40:41.481 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 7.4 M 2024-04-16T07:40:41.481 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-16T07:40:41.481 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T07:40:41.481 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T07:40:41.481 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T07:40:41.481 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T07:40:41.481 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T07:40:41.481 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T07:40:41.481 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T07:40:41.481 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T07:40:41.482 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T07:40:41.482 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T07:40:41.482 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T07:40:41.482 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:41.482 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-16T07:40:41.482 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2024-04-16T07:40:41.482 INFO:teuthology.orchestra.run.smithi031.stdout:Install 12 Packages 2024-04-16T07:40:41.482 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:41.483 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 37 M 2024-04-16T07:40:41.483 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 209 M 2024-04-16T07:40:41.483 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-16T07:40:41.709 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:58 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:40:41.813 INFO:teuthology.orchestra.run.smithi031.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 235 kB/s | 32 kB 00:00 2024-04-16T07:40:41.855 INFO:teuthology.orchestra.run.smithi031.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 412 kB/s | 17 kB 00:00 2024-04-16T07:40:41.867 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:40:41.867 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================== 2024-04-16T07:40:41.868 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-04-16T07:40:41.868 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================== 2024-04-16T07:40:41.868 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-16T07:40:41.868 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 3.5 M 2024-04-16T07:40:41.868 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-04-16T07:40:41.868 INFO:teuthology.orchestra.run.smithi094.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-16T07:40:41.868 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 34 k 2024-04-16T07:40:41.868 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 16 k 2024-04-16T07:40:41.868 INFO:teuthology.orchestra.run.smithi094.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-16T07:40:41.868 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-16T07:40:41.868 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-16T07:40:41.868 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-16T07:40:41.868 INFO:teuthology.orchestra.run.smithi094.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-16T07:40:41.869 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T07:40:41.869 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T07:40:41.869 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:41.869 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-16T07:40:41.869 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================== 2024-04-16T07:40:41.869 INFO:teuthology.orchestra.run.smithi094.stdout:Install 11 Packages 2024-04-16T07:40:41.869 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:41.869 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 7.6 M 2024-04-16T07:40:41.870 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 100 M 2024-04-16T07:40:41.870 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-16T07:40:41.930 INFO:teuthology.orchestra.run.smithi031.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 456 kB/s | 34 kB 00:00 2024-04-16T07:40:42.147 INFO:teuthology.orchestra.run.smithi031.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-04-16T07:40:42.314 INFO:teuthology.orchestra.run.smithi031.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-04-16T07:40:42.364 INFO:teuthology.orchestra.run.smithi031.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.3 MB/s | 217 kB 00:00 2024-04-16T07:40:42.624 INFO:teuthology.orchestra.run.smithi094.stdout:(1/11): ceph-prometheus-alerts-19.0.0-3109.g30b 41 kB/s | 16 kB 00:00 2024-04-16T07:40:42.790 INFO:teuthology.orchestra.run.smithi094.stdout:(2/11): ceph-grafana-dashboards-19.0.0-3109.g30 60 kB/s | 34 kB 00:00 2024-04-16T07:40:42.849 INFO:teuthology.orchestra.run.smithi094.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.0 MB/s | 238 kB 00:00 2024-04-16T07:40:42.881 INFO:teuthology.orchestra.run.smithi031.stdout:(7/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-04-16T07:40:43.040 INFO:teuthology.orchestra.run.smithi031.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.2 MB/s | 3.0 MB 00:01 2024-04-16T07:40:43.223 INFO:teuthology.orchestra.run.smithi031.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-04-16T07:40:43.249 INFO:teuthology.orchestra.run.smithi094.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 1.8 MB/s | 863 kB 00:00 2024-04-16T07:40:43.257 INFO:teuthology.orchestra.run.smithi031.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-04-16T07:40:43.408 INFO:teuthology.orchestra.run.smithi094.stdout:(5/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 981 kB/s | 548 kB 00:00 2024-04-16T07:40:43.508 INFO:teuthology.orchestra.run.smithi094.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-04-16T07:40:43.643 INFO:teuthology.orchestra.run.smithi094.stdout:(7/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 8.7 MB/s | 2.0 MB 00:00 2024-04-16T07:40:43.742 INFO:teuthology.orchestra.run.smithi094.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 308 kB/s | 31 kB 00:00 2024-04-16T07:40:43.768 INFO:teuthology.orchestra.run.smithi094.stdout:(9/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 557 kB/s | 144 kB 00:00 2024-04-16T07:40:43.799 INFO:teuthology.orchestra.run.smithi031.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 427 kB/s | 245 kB 00:00 2024-04-16T07:40:43.835 INFO:teuthology.orchestra.run.smithi094.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-04-16T07:40:45.335 INFO:teuthology.orchestra.run.smithi094.stdout:(11/11): ceph-mgr-dashboard-19.0.0-3109.g30b0aa 1.1 MB/s | 3.5 MB 00:03 2024-04-16T07:40:45.340 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:40:45.340 INFO:teuthology.orchestra.run.smithi094.stdout:Total 2.2 MB/s | 7.6 MB 00:03 2024-04-16T07:40:45.361 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-16T07:40:45.380 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-16T07:40:45.380 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-16T07:40:45.567 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-16T07:40:45.567 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-16T07:40:46.054 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-16T07:40:46.133 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-16T07:40:46.171 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-16T07:40:46.260 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-16T07:40:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-16T07:40:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-16T07:40:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-16T07:40:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-16T07:40:46.622 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-16T07:40:46.654 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9 9/11 2024-04-16T07:40:47.697 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el 10/11 2024-04-16T07:40:47.738 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 11/11 2024-04-16T07:40:49.042 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 11/11 2024-04-16T07:40:49.043 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el 1/11 2024-04-16T07:40:49.043 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 2/11 2024-04-16T07:40:49.043 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9 3/11 2024-04-16T07:40:49.043 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-16T07:40:49.043 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-16T07:40:49.043 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-16T07:40:49.043 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-16T07:40:49.043 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-16T07:40:49.043 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-16T07:40:49.043 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-16T07:40:49.377 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-16T07:40:49.377 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:49.377 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-16T07:40:49.377 INFO:teuthology.orchestra.run.smithi094.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-16T07:40:49.377 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:40:49.377 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:40:49.377 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:40:49.377 INFO:teuthology.orchestra.run.smithi094.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-16T07:40:49.377 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-16T07:40:49.377 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-16T07:40:49.377 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-16T07:40:49.377 INFO:teuthology.orchestra.run.smithi094.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-16T07:40:49.378 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T07:40:49.378 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T07:40:49.378 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:49.378 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:40:49.647 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T07:40:49.840 INFO:teuthology.orchestra.run.smithi031.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-3 928 kB/s | 7.4 MB 00:08 2024-04-16T07:40:49.842 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:40:49.843 INFO:teuthology.orchestra.run.smithi031.stdout:Total 4.4 MB/s | 37 MB 00:08 2024-04-16T07:40:49.889 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-16T07:40:49.915 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-16T07:40:49.915 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-16T07:40:50.225 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:04:07 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:40:50.288 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-16T07:40:50.288 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-16T07:40:50.379 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:40:50.380 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2024-04-16T07:40:50.380 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-04-16T07:40:50.380 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2024-04-16T07:40:50.380 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-16T07:40:50.380 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 7.4 M 2024-04-16T07:40:50.380 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-04-16T07:40:50.380 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T07:40:50.380 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T07:40:50.380 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T07:40:50.380 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T07:40:50.380 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T07:40:50.381 INFO:teuthology.orchestra.run.smithi094.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T07:40:50.381 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T07:40:50.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T07:40:50.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T07:40:50.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T07:40:50.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T07:40:50.381 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:50.381 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-16T07:40:50.381 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2024-04-16T07:40:50.381 INFO:teuthology.orchestra.run.smithi094.stdout:Install 12 Packages 2024-04-16T07:40:50.381 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:40:50.382 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 37 M 2024-04-16T07:40:50.382 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 209 M 2024-04-16T07:40:50.382 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-16T07:40:50.959 INFO:teuthology.orchestra.run.smithi094.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 254 kB/s | 32 kB 00:00 2024-04-16T07:40:51.001 INFO:teuthology.orchestra.run.smithi094.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 412 kB/s | 17 kB 00:00 2024-04-16T07:40:51.011 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-16T07:40:51.043 INFO:teuthology.orchestra.run.smithi094.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 837 kB/s | 34 kB 00:00 2024-04-16T07:40:51.097 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T07:40:51.128 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T07:40:51.226 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:40:51.269 INFO:teuthology.orchestra.run.smithi094.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-04-16T07:40:51.273 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T07:40:51.444 INFO:teuthology.orchestra.run.smithi094.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-04-16T07:40:51.457 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:40:51.486 INFO:teuthology.orchestra.run.smithi094.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-04-16T07:40:51.488 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:40:51.545 INFO:teuthology.orchestra.run.smithi094.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.2 MB/s | 3.0 MB 00:00 2024-04-16T07:40:51.567 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T07:40:51.761 INFO:teuthology.orchestra.run.smithi094.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.7 MB/s | 803 kB 00:00 2024-04-16T07:40:51.837 INFO:teuthology.orchestra.run.smithi094.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 195 kB 00:00 2024-04-16T07:40:52.005 INFO:teuthology.orchestra.run.smithi094.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-04-16T07:40:52.196 INFO:teuthology.orchestra.run.smithi094.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 685 kB/s | 245 kB 00:00 2024-04-16T07:40:52.256 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T07:40:52.371 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T07:40:53.669 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T07:40:54.749 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T07:40:54.778 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 12/12 2024-04-16T07:40:57.211 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 12/12 2024-04-16T07:40:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 1/12 2024-04-16T07:40:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T07:40:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:40:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T07:40:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:40:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:40:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T07:40:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T07:40:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T07:40:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T07:40:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T07:40:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T07:40:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:57.649 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-16T07:40:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:40:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T07:40:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T07:40:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T07:40:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T07:40:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T07:40:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T07:40:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T07:40:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T07:40:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T07:40:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T07:40:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T07:40:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:57.650 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:40:58.042 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-rook 2024-04-16T07:40:58.633 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:09 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:40:58.793 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:40:58.794 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2024-04-16T07:40:58.794 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-16T07:40:58.794 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2024-04-16T07:40:58.794 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-16T07:40:58.794 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 49 k 2024-04-16T07:40:58.794 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-16T07:40:58.794 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T07:40:58.794 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T07:40:58.794 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T07:40:58.795 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T07:40:58.795 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T07:40:58.795 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T07:40:58.795 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T07:40:58.795 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T07:40:58.795 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T07:40:58.795 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T07:40:58.795 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T07:40:58.795 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T07:40:58.795 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:58.795 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-16T07:40:58.795 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2024-04-16T07:40:58.795 INFO:teuthology.orchestra.run.smithi031.stdout:Install 13 Packages 2024-04-16T07:40:58.795 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:40:58.796 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 2.0 M 2024-04-16T07:40:58.796 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 26 M 2024-04-16T07:40:58.796 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-16T07:40:58.986 INFO:teuthology.orchestra.run.smithi031.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 239 kB/s | 24 kB 00:00 2024-04-16T07:40:59.036 INFO:teuthology.orchestra.run.smithi031.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 119 kB/s | 18 kB 00:00 2024-04-16T07:40:59.077 INFO:teuthology.orchestra.run.smithi031.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-04-16T07:40:59.103 INFO:teuthology.orchestra.run.smithi031.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-04-16T07:40:59.128 INFO:teuthology.orchestra.run.smithi031.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-04-16T07:40:59.153 INFO:teuthology.orchestra.run.smithi031.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 872 kB/s | 43 kB 00:00 2024-04-16T07:40:59.412 INFO:teuthology.orchestra.run.smithi031.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 114 kB/s | 32 kB 00:00 2024-04-16T07:40:59.437 INFO:teuthology.orchestra.run.smithi031.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 53 kB/s | 15 kB 00:00 2024-04-16T07:40:59.463 INFO:teuthology.orchestra.run.smithi031.stdout:(9/13): ceph-mgr-rook-19.0.0-3109.g30b0aa00.el9 85 kB/s | 49 kB 00:00 2024-04-16T07:40:59.679 INFO:teuthology.orchestra.run.smithi031.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 758 kB/s | 203 kB 00:00 2024-04-16T07:40:59.788 INFO:teuthology.orchestra.run.smithi031.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 829 kB/s | 90 kB 00:00 2024-04-16T07:40:59.830 INFO:teuthology.orchestra.run.smithi031.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 161 kB/s | 59 kB 00:00 2024-04-16T07:41:00.022 INFO:teuthology.orchestra.run.smithi031.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-04-16T07:41:00.026 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:00.026 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-04-16T07:41:00.026 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-16T07:41:00.041 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-16T07:41:00.041 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-16T07:41:00.139 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-16T07:41:00.140 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-16T07:41:00.660 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-16T07:41:00.704 INFO:teuthology.orchestra.run.smithi094.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-3 767 kB/s | 7.4 MB 00:09 2024-04-16T07:41:00.706 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:00.706 INFO:teuthology.orchestra.run.smithi094.stdout:Total 3.6 MB/s | 37 MB 00:10 2024-04-16T07:41:00.751 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-16T07:41:00.777 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-16T07:41:00.777 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-16T07:41:00.835 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T07:41:00.893 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T07:41:00.948 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T07:41:00.984 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T07:41:01.026 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T07:41:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T07:41:01.137 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-16T07:41:01.137 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-16T07:41:01.232 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T07:41:01.286 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T07:41:01.854 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-16T07:41:01.918 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T07:41:01.950 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T07:41:02.043 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:41:02.052 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T07:41:02.095 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T07:41:02.099 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T07:41:02.133 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T07:41:02.181 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T07:41:02.210 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 13/13 2024-04-16T07:41:02.272 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:41:02.302 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:41:02.378 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T07:41:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 13/13 2024-04-16T07:41:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 1/13 2024-04-16T07:41:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T07:41:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T07:41:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T07:41:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T07:41:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T07:41:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T07:41:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T07:41:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T07:41:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T07:41:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T07:41:02.992 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T07:41:03.041 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T07:41:03.165 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T07:41:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T07:41:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:03.362 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-16T07:41:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:41:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T07:41:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T07:41:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T07:41:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T07:41:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T07:41:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T07:41:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T07:41:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T07:41:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T07:41:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T07:41:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T07:41:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T07:41:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:03.363 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:41:03.586 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T07:41:04.184 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:15 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:41:04.345 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:41:04.345 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2024-04-16T07:41:04.345 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-16T07:41:04.345 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2024-04-16T07:41:04.346 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-16T07:41:04.346 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 142 k 2024-04-16T07:41:04.346 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-16T07:41:04.346 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T07:41:04.346 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T07:41:04.346 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T07:41:04.346 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T07:41:04.346 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T07:41:04.346 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:04.346 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-16T07:41:04.346 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2024-04-16T07:41:04.346 INFO:teuthology.orchestra.run.smithi031.stdout:Install 6 Packages 2024-04-16T07:41:04.346 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:04.347 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 6.8 M 2024-04-16T07:41:04.347 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 33 M 2024-04-16T07:41:04.347 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-16T07:41:04.374 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T07:41:04.900 INFO:teuthology.orchestra.run.smithi031.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-04-16T07:41:05.150 INFO:teuthology.orchestra.run.smithi031.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-04-16T07:41:05.258 INFO:teuthology.orchestra.run.smithi031.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-04-16T07:41:05.284 INFO:teuthology.orchestra.run.smithi031.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 431 kB/s | 58 kB 00:00 2024-04-16T07:41:05.309 INFO:teuthology.orchestra.run.smithi031.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-04-16T07:41:05.407 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T07:41:05.444 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 12/12 2024-04-16T07:41:05.576 INFO:teuthology.orchestra.run.smithi031.stdout:(6/6): ceph-mgr-cephadm-19.0.0-3109.g30b0aa00.e 167 kB/s | 142 kB 00:00 2024-04-16T07:41:05.577 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:05.578 INFO:teuthology.orchestra.run.smithi031.stdout:Total 5.5 MB/s | 6.8 MB 00:01 2024-04-16T07:41:05.591 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-16T07:41:05.602 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-16T07:41:05.602 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-16T07:41:05.696 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-16T07:41:05.696 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-16T07:41:05.994 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-16T07:41:06.173 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T07:41:06.220 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T07:41:06.768 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T07:41:06.859 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T07:41:06.945 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T07:41:06.972 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 6/6 2024-04-16T07:41:07.752 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 6/6 2024-04-16T07:41:07.752 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/6 2024-04-16T07:41:07.752 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T07:41:07.752 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T07:41:07.752 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T07:41:07.753 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T07:41:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 12/12 2024-04-16T07:41:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 1/12 2024-04-16T07:41:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T07:41:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:41:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T07:41:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:41:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:41:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T07:41:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T07:41:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T07:41:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T07:41:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T07:41:08.062 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T07:41:08.062 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:08.062 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-16T07:41:08.062 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:41:08.062 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T07:41:08.063 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T07:41:08.063 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T07:41:08.063 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T07:41:08.063 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T07:41:08.063 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:08.063 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:41:08.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-fuse 2024-04-16T07:41:08.438 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T07:41:08.438 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:08.438 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-16T07:41:08.439 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:41:08.439 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T07:41:08.439 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T07:41:08.439 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T07:41:08.439 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T07:41:08.439 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T07:41:08.439 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T07:41:08.439 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T07:41:08.439 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T07:41:08.439 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T07:41:08.439 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T07:41:08.439 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T07:41:08.439 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:08.439 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:41:08.925 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:19 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:41:08.938 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-rook 2024-04-16T07:41:09.085 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:41:09.086 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:09.086 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-16T07:41:09.086 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:09.086 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-16T07:41:09.086 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 871 k 2024-04-16T07:41:09.087 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-16T07:41:09.087 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T07:41:09.087 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:09.087 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-16T07:41:09.087 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:09.087 INFO:teuthology.orchestra.run.smithi031.stdout:Install 2 Packages 2024-04-16T07:41:09.087 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:09.087 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 950 k 2024-04-16T07:41:09.087 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 4.0 M 2024-04-16T07:41:09.087 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-16T07:41:09.210 INFO:teuthology.orchestra.run.smithi031.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 640 kB/s | 79 kB 00:00 2024-04-16T07:41:09.528 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:04:26 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:41:09.680 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:41:09.681 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-04-16T07:41:09.682 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-04-16T07:41:09.682 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-04-16T07:41:09.682 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-16T07:41:09.682 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 49 k 2024-04-16T07:41:09.682 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-04-16T07:41:09.682 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T07:41:09.682 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T07:41:09.682 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T07:41:09.682 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T07:41:09.682 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T07:41:09.682 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T07:41:09.683 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T07:41:09.683 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T07:41:09.683 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T07:41:09.683 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T07:41:09.683 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T07:41:09.683 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T07:41:09.683 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:09.683 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-16T07:41:09.683 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-04-16T07:41:09.683 INFO:teuthology.orchestra.run.smithi094.stdout:Install 13 Packages 2024-04-16T07:41:09.683 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:09.684 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 2.0 M 2024-04-16T07:41:09.684 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 26 M 2024-04-16T07:41:09.684 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-16T07:41:09.955 INFO:teuthology.orchestra.run.smithi094.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 193 kB/s | 24 kB 00:00 2024-04-16T07:41:09.980 INFO:teuthology.orchestra.run.smithi094.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 119 kB/s | 18 kB 00:00 2024-04-16T07:41:10.022 INFO:teuthology.orchestra.run.smithi094.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-04-16T07:41:10.048 INFO:teuthology.orchestra.run.smithi094.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-04-16T07:41:10.081 INFO:teuthology.orchestra.run.smithi094.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-04-16T07:41:10.107 INFO:teuthology.orchestra.run.smithi094.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 729 kB/s | 43 kB 00:00 2024-04-16T07:41:10.340 INFO:teuthology.orchestra.run.smithi094.stdout:(7/13): ceph-mgr-rook-19.0.0-3109.g30b0aa00.el9 96 kB/s | 49 kB 00:00 2024-04-16T07:41:10.466 INFO:teuthology.orchestra.run.smithi094.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 84 kB/s | 32 kB 00:00 2024-04-16T07:41:10.491 INFO:teuthology.orchestra.run.smithi094.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 39 kB/s | 15 kB 00:00 2024-04-16T07:41:10.691 INFO:teuthology.orchestra.run.smithi094.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 295 kB/s | 59 kB 00:00 2024-04-16T07:41:10.866 INFO:teuthology.orchestra.run.smithi094.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 385 kB/s | 203 kB 00:00 2024-04-16T07:41:10.892 INFO:teuthology.orchestra.run.smithi094.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 448 kB/s | 90 kB 00:00 2024-04-16T07:41:10.959 INFO:teuthology.orchestra.run.smithi094.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.1 MB/s | 1.0 MB 00:00 2024-04-16T07:41:10.963 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:10.963 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-04-16T07:41:10.963 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-16T07:41:10.976 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-16T07:41:10.976 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-16T07:41:11.065 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-16T07:41:11.066 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-16T07:41:11.583 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-16T07:41:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T07:41:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T07:41:11.848 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T07:41:11.886 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T07:41:11.927 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T07:41:12.021 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T07:41:12.129 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T07:41:12.187 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T07:41:12.905 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T07:41:12.959 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T07:41:12.993 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T07:41:13.041 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T07:41:13.063 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 13/13 2024-04-16T07:41:13.909 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 13/13 2024-04-16T07:41:13.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 1/13 2024-04-16T07:41:13.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T07:41:13.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T07:41:13.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T07:41:13.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T07:41:13.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T07:41:13.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T07:41:13.910 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T07:41:13.910 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T07:41:13.910 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T07:41:13.910 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T07:41:13.910 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T07:41:14.275 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T07:41:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:14.276 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-16T07:41:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:41:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T07:41:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T07:41:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T07:41:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T07:41:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T07:41:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T07:41:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T07:41:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T07:41:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T07:41:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T07:41:14.277 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T07:41:14.277 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T07:41:14.277 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:14.277 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:41:14.536 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T07:41:15.018 INFO:teuthology.orchestra.run.smithi031.stdout:(2/2): ceph-fuse-19.0.0-3109.g30b0aa00.el9.x86_ 147 kB/s | 871 kB 00:05 2024-04-16T07:41:15.020 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:15.020 INFO:teuthology.orchestra.run.smithi031.stdout:Total 160 kB/s | 950 kB 00:05 2024-04-16T07:41:15.020 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-16T07:41:15.050 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-16T07:41:15.050 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-16T07:41:15.118 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:04:32 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:41:15.127 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-16T07:41:15.127 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-16T07:41:15.271 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:41:15.272 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-04-16T07:41:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-04-16T07:41:15.272 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-04-16T07:41:15.272 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-16T07:41:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 142 k 2024-04-16T07:41:15.272 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-04-16T07:41:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T07:41:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T07:41:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T07:41:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T07:41:15.273 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T07:41:15.273 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:15.273 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-16T07:41:15.273 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-04-16T07:41:15.273 INFO:teuthology.orchestra.run.smithi094.stdout:Install 6 Packages 2024-04-16T07:41:15.273 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:15.274 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 6.8 M 2024-04-16T07:41:15.274 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 33 M 2024-04-16T07:41:15.274 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-16T07:41:15.309 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-16T07:41:15.441 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T07:41:15.512 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/2 2024-04-16T07:41:15.771 INFO:teuthology.orchestra.run.smithi094.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-04-16T07:41:15.854 INFO:teuthology.orchestra.run.smithi094.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-04-16T07:41:15.929 INFO:teuthology.orchestra.run.smithi094.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 780 kB/s | 58 kB 00:00 2024-04-16T07:41:15.971 INFO:teuthology.orchestra.run.smithi094.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.2 MB/s | 51 kB 00:00 2024-04-16T07:41:15.996 INFO:teuthology.orchestra.run.smithi094.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.4 MB/s | 548 kB 00:00 2024-04-16T07:41:16.177 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/2 2024-04-16T07:41:16.178 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/2 2024-04-16T07:41:16.501 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T07:41:16.501 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:16.501 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-16T07:41:16.501 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T07:41:16.501 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:16.501 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:41:16.805 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-volume 2024-04-16T07:41:16.872 INFO:teuthology.orchestra.run.smithi094.stdout:(6/6): ceph-mgr-cephadm-19.0.0-3109.g30b0aa00.e 109 kB/s | 142 kB 00:01 2024-04-16T07:41:16.873 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:16.874 INFO:teuthology.orchestra.run.smithi094.stdout:Total 4.2 MB/s | 6.8 MB 00:01 2024-04-16T07:41:16.888 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-16T07:41:16.899 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-16T07:41:16.899 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-16T07:41:16.988 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-16T07:41:16.990 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-16T07:41:17.279 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-16T07:41:17.403 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:28 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:41:17.453 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T07:41:17.507 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T07:41:17.563 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:41:17.563 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:17.563 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-16T07:41:17.563 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:17.563 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-16T07:41:17.564 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 273 k 2024-04-16T07:41:17.564 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-16T07:41:17.564 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-16T07:41:17.564 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:17.564 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-16T07:41:17.564 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:17.564 INFO:teuthology.orchestra.run.smithi031.stdout:Install 2 Packages 2024-04-16T07:41:17.564 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:17.565 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 342 k 2024-04-16T07:41:17.565 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 1.5 M 2024-04-16T07:41:17.565 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-16T07:41:17.722 INFO:teuthology.orchestra.run.smithi031.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 438 kB/s | 69 kB 00:00 2024-04-16T07:41:18.014 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T07:41:18.101 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T07:41:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T07:41:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 6/6 2024-04-16T07:41:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 6/6 2024-04-16T07:41:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/6 2024-04-16T07:41:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T07:41:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T07:41:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T07:41:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T07:41:19.314 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T07:41:19.315 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:19.315 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-16T07:41:19.315 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:41:19.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T07:41:19.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T07:41:19.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T07:41:19.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T07:41:19.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T07:41:19.315 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:19.315 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:41:19.497 INFO:teuthology.orchestra.run.smithi031.stdout:(2/2): ceph-volume-19.0.0-3109.g30b0aa00.el9.no 141 kB/s | 273 kB 00:01 2024-04-16T07:41:19.498 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:19.499 INFO:teuthology.orchestra.run.smithi031.stdout:Total 177 kB/s | 342 kB 00:01 2024-04-16T07:41:19.499 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-16T07:41:19.510 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-16T07:41:19.510 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-16T07:41:19.553 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-16T07:41:19.553 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-16T07:41:19.597 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-fuse 2024-04-16T07:41:19.782 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-16T07:41:19.968 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-16T07:41:20.008 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2/2 2024-04-16T07:41:20.024 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2/2 2024-04-16T07:41:20.025 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:41:20.025 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-16T07:41:20.025 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:20.175 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:04:37 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:41:20.327 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:41:20.328 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:20.328 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-04-16T07:41:20.329 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:20.329 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-16T07:41:20.329 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 871 k 2024-04-16T07:41:20.329 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-04-16T07:41:20.329 INFO:teuthology.orchestra.run.smithi094.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T07:41:20.329 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:20.329 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-16T07:41:20.330 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:20.330 INFO:teuthology.orchestra.run.smithi094.stdout:Install 2 Packages 2024-04-16T07:41:20.330 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:20.330 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 950 k 2024-04-16T07:41:20.330 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 4.0 M 2024-04-16T07:41:20.330 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-16T07:41:20.492 INFO:teuthology.orchestra.run.smithi094.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 484 kB/s | 79 kB 00:00 2024-04-16T07:41:20.867 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 1/2 2024-04-16T07:41:21.223 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-16T07:41:21.223 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:21.223 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-16T07:41:21.223 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:41:21.223 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-16T07:41:21.223 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:21.223 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:41:21.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados-devel 2024-04-16T07:41:22.146 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:33 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:41:22.192 INFO:teuthology.orchestra.run.smithi094.stdout:(2/2): ceph-fuse-19.0.0-3109.g30b0aa00.el9.x86_ 468 kB/s | 871 kB 00:01 2024-04-16T07:41:22.193 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:22.193 INFO:teuthology.orchestra.run.smithi094.stdout:Total 509 kB/s | 950 kB 00:01 2024-04-16T07:41:22.194 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-16T07:41:22.219 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-16T07:41:22.219 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-16T07:41:22.296 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-16T07:41:22.296 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-16T07:41:22.307 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:41:22.307 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:22.308 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-04-16T07:41:22.308 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:22.308 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-16T07:41:22.308 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 125 k 2024-04-16T07:41:22.308 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:22.308 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-16T07:41:22.308 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:22.308 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-04-16T07:41:22.308 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:22.308 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 125 k 2024-04-16T07:41:22.308 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 445 k 2024-04-16T07:41:22.308 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-16T07:41:22.486 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-16T07:41:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T07:41:22.707 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/2 2024-04-16T07:41:22.911 INFO:teuthology.orchestra.run.smithi031.stdout:librados-devel-19.0.0-3109.g30b0aa00.el9.x86_64 207 kB/s | 125 kB 00:00 2024-04-16T07:41:22.911 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:22.911 INFO:teuthology.orchestra.run.smithi031.stdout:Total 207 kB/s | 125 kB 00:00 2024-04-16T07:41:22.912 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-16T07:41:22.918 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-16T07:41:22.918 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-16T07:41:22.977 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-16T07:41:22.978 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-16T07:41:23.125 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-16T07:41:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:23.375 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/2 2024-04-16T07:41:23.376 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/2 2024-04-16T07:41:23.713 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T07:41:23.713 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:23.713 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-16T07:41:23.714 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T07:41:23.714 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:23.714 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:41:23.714 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:23.987 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-volume 2024-04-16T07:41:23.998 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:23.999 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:23.999 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-16T07:41:23.999 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:41:23.999 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:23.999 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:41:24.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs2 2024-04-16T07:41:24.587 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:04:41 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:41:24.744 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:41:24.745 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:24.745 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-04-16T07:41:24.745 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:24.745 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-16T07:41:24.745 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 273 k 2024-04-16T07:41:24.745 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-04-16T07:41:24.745 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-16T07:41:24.745 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:24.746 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-16T07:41:24.746 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:24.746 INFO:teuthology.orchestra.run.smithi094.stdout:Install 2 Packages 2024-04-16T07:41:24.746 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:24.746 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 342 k 2024-04-16T07:41:24.746 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 1.5 M 2024-04-16T07:41:24.747 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-16T07:41:24.821 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:35 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:41:24.855 INFO:teuthology.orchestra.run.smithi094.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 631 kB/s | 69 kB 00:00 2024-04-16T07:41:24.924 INFO:teuthology.orchestra.run.smithi031.stdout:Package libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:41:24.979 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:41:24.979 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-16T07:41:24.980 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:41:25.058 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs-devel 2024-04-16T07:41:25.654 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:36 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:41:25.811 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:41:25.812 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:25.812 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-04-16T07:41:25.812 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:25.813 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-16T07:41:25.813 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 32 k 2024-04-16T07:41:25.813 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:25.813 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-16T07:41:25.813 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:25.813 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-04-16T07:41:25.813 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:25.813 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 32 k 2024-04-16T07:41:25.813 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 142 k 2024-04-16T07:41:25.813 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-16T07:41:26.264 INFO:teuthology.orchestra.run.smithi094.stdout:(2/2): ceph-volume-19.0.0-3109.g30b0aa00.el9.no 180 kB/s | 273 kB 00:01 2024-04-16T07:41:26.266 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:26.266 INFO:teuthology.orchestra.run.smithi094.stdout:Total 225 kB/s | 342 kB 00:01 2024-04-16T07:41:26.266 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-16T07:41:26.276 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-16T07:41:26.276 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-16T07:41:26.316 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-16T07:41:26.316 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-16T07:41:26.338 INFO:teuthology.orchestra.run.smithi031.stdout:libcephfs-devel-19.0.0-3109.g30b0aa00.el9.x86_6 60 kB/s | 32 kB 00:00 2024-04-16T07:41:26.339 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:26.339 INFO:teuthology.orchestra.run.smithi031.stdout:Total 60 kB/s | 32 kB 00:00 2024-04-16T07:41:26.339 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-16T07:41:26.343 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-16T07:41:26.343 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-16T07:41:26.372 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-16T07:41:26.372 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-16T07:41:26.464 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-16T07:41:26.557 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-16T07:41:26.573 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:26.754 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-16T07:41:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2/2 2024-04-16T07:41:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2/2 2024-04-16T07:41:26.808 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:41:26.808 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-16T07:41:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:26.947 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:27.204 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:27.204 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:27.204 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-16T07:41:27.204 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:41:27.204 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:27.205 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:41:27.379 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados2 2024-04-16T07:41:27.590 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 1/2 2024-04-16T07:41:27.923 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-16T07:41:27.923 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:27.923 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-16T07:41:27.923 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:41:27.923 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-16T07:41:27.923 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:27.923 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:41:27.971 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:38 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:41:28.075 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:41:28.130 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:41:28.132 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-16T07:41:28.132 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:41:28.211 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librbd1 2024-04-16T07:41:28.234 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados-devel 2024-04-16T07:41:28.802 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:39 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:41:28.833 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:04:45 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:41:28.905 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:41:28.961 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:41:28.961 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-16T07:41:28.962 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:41:28.985 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:41:28.985 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:28.985 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-04-16T07:41:28.985 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:28.985 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-16T07:41:28.986 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 125 k 2024-04-16T07:41:28.986 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:28.986 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-16T07:41:28.986 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:28.986 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-04-16T07:41:28.986 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:28.987 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 125 k 2024-04-16T07:41:28.987 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 445 k 2024-04-16T07:41:28.987 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-16T07:41:29.049 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rados 2024-04-16T07:41:29.646 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:40 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:41:29.655 INFO:teuthology.orchestra.run.smithi094.stdout:librados-devel-19.0.0-3109.g30b0aa00.el9.x86_64 187 kB/s | 125 kB 00:00 2024-04-16T07:41:29.655 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:29.656 INFO:teuthology.orchestra.run.smithi094.stdout:Total 186 kB/s | 125 kB 00:00 2024-04-16T07:41:29.656 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-16T07:41:29.662 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-16T07:41:29.662 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-16T07:41:29.718 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-16T07:41:29.719 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-16T07:41:29.749 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:41:29.806 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:41:29.870 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-16T07:41:29.870 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:41:29.872 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-16T07:41:29.886 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rgw 2024-04-16T07:41:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:30.448 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:30.479 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:41 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:41:30.581 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:41:30.636 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:41:30.637 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-16T07:41:30.637 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:41:30.721 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:30.721 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:30.721 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-16T07:41:30.721 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:41:30.721 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:30.721 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:41:30.782 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-cephfs 2024-04-16T07:41:30.991 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs2 2024-04-16T07:41:31.377 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:42 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:41:31.479 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:41:31.533 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:41:31.534 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-16T07:41:31.534 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:41:31.591 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:04:48 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:41:31.616 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rbd 2024-04-16T07:41:31.695 INFO:teuthology.orchestra.run.smithi094.stdout:Package libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:41:31.749 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:41:31.750 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-04-16T07:41:31.750 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:41:31.829 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs-devel 2024-04-16T07:41:32.206 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:43 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:41:32.309 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:41:32.364 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:41:32.365 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-16T07:41:32.365 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:41:32.409 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:04:49 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:41:32.452 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-fuse 2024-04-16T07:41:32.559 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:41:32.559 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:32.560 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-04-16T07:41:32.560 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:32.560 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-16T07:41:32.560 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 32 k 2024-04-16T07:41:32.560 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:32.560 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-16T07:41:32.560 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:32.560 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-04-16T07:41:32.560 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:32.560 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 32 k 2024-04-16T07:41:32.560 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 142 k 2024-04-16T07:41:32.560 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-16T07:41:33.046 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:44 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:41:33.075 INFO:teuthology.orchestra.run.smithi094.stdout:libcephfs-devel-19.0.0-3109.g30b0aa00.el9.x86_6 61 kB/s | 32 kB 00:00 2024-04-16T07:41:33.076 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:33.076 INFO:teuthology.orchestra.run.smithi094.stdout:Total 61 kB/s | 32 kB 00:00 2024-04-16T07:41:33.077 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-16T07:41:33.080 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-16T07:41:33.080 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-16T07:41:33.102 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-16T07:41:33.102 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-16T07:41:33.195 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-16T07:41:33.204 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:41:33.204 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:33.204 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-04-16T07:41:33.204 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:33.204 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-16T07:41:33.204 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 90 k 2024-04-16T07:41:33.205 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:33.205 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-16T07:41:33.205 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:33.205 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-04-16T07:41:33.205 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:33.205 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 90 k 2024-04-16T07:41:33.205 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 313 k 2024-04-16T07:41:33.206 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-16T07:41:33.291 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:33.651 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:33.883 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-fuse-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 132 kB/s | 90 kB 00:00 2024-04-16T07:41:33.884 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:33.884 INFO:teuthology.orchestra.run.smithi031.stdout:Total 132 kB/s | 90 kB 00:00 2024-04-16T07:41:33.885 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-16T07:41:33.903 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-16T07:41:33.903 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-16T07:41:33.921 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:33.922 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:33.922 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-16T07:41:33.922 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:41:33.922 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:33.922 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:41:33.967 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-16T07:41:33.968 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-16T07:41:34.118 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados2 2024-04-16T07:41:34.122 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-16T07:41:34.236 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:34.696 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:04:51 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:41:34.796 INFO:teuthology.orchestra.run.smithi094.stdout:Package librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:41:34.848 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:41:34.849 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-04-16T07:41:34.850 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:41:34.932 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librbd1 2024-04-16T07:41:35.047 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:35.047 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:35.048 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-16T07:41:35.048 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:41:35.048 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:35.048 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:41:35.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-mirror 2024-04-16T07:41:35.513 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:04:52 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:41:35.613 INFO:teuthology.orchestra.run.smithi094.stdout:Package librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:41:35.665 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:41:35.667 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-04-16T07:41:35.667 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:41:35.749 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rados 2024-04-16T07:41:35.894 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:46 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:41:36.053 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:41:36.054 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:36.054 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-04-16T07:41:36.054 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:36.054 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-16T07:41:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.4 M 2024-04-16T07:41:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:36.055 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-16T07:41:36.055 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:36.055 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-04-16T07:41:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:36.055 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 3.4 M 2024-04-16T07:41:36.055 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 29 M 2024-04-16T07:41:36.055 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-16T07:41:36.333 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:04:53 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:41:36.432 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:41:36.484 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:41:36.485 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-04-16T07:41:36.485 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:41:36.562 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rgw 2024-04-16T07:41:37.144 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:04:54 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:41:37.245 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:41:37.296 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:41:37.297 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-04-16T07:41:37.297 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:41:37.382 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-cephfs 2024-04-16T07:41:37.964 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:04:54 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:41:38.063 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:41:38.114 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:41:38.115 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-04-16T07:41:38.115 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:41:38.207 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rbd 2024-04-16T07:41:38.785 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:04:55 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:41:38.884 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:41:38.934 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:41:38.936 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-04-16T07:41:38.936 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:41:39.014 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-fuse 2024-04-16T07:41:39.595 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:04:56 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:41:39.746 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:41:39.747 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-04-16T07:41:39.747 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:39.748 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-16T07:41:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 90 k 2024-04-16T07:41:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:39.748 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-16T07:41:39.748 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:39.748 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-04-16T07:41:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:39.749 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 90 k 2024-04-16T07:41:39.749 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 313 k 2024-04-16T07:41:39.749 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-16T07:41:40.621 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-fuse-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 103 kB/s | 90 kB 00:00 2024-04-16T07:41:40.621 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:40.621 INFO:teuthology.orchestra.run.smithi094.stdout:Total 102 kB/s | 90 kB 00:00 2024-04-16T07:41:40.622 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-16T07:41:40.632 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-16T07:41:40.632 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-16T07:41:40.693 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-16T07:41:40.693 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-16T07:41:40.846 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-16T07:41:40.944 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:41.323 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:41.586 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:41.587 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:41.587 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-16T07:41:41.587 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:41:41.587 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:41.587 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:41:41.835 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-mirror 2024-04-16T07:41:42.427 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:04:59 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:41:42.579 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:41:42.580 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-04-16T07:41:42.580 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:42.580 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-16T07:41:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.4 M 2024-04-16T07:41:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:42.581 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-16T07:41:42.581 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:42.581 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-04-16T07:41:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:42.581 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 3.4 M 2024-04-16T07:41:42.581 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 29 M 2024-04-16T07:41:42.581 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-16T07:41:42.868 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-mirror-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 511 kB/s | 3.4 MB 00:06 2024-04-16T07:41:42.869 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:42.870 INFO:teuthology.orchestra.run.smithi031.stdout:Total 511 kB/s | 3.4 MB 00:06 2024-04-16T07:41:42.870 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-16T07:41:42.884 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-16T07:41:42.884 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-16T07:41:42.967 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-16T07:41:42.967 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-16T07:41:43.554 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-16T07:41:43.619 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:43.645 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:43.645 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:41:43.645 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T07:41:43.645 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:41:43.645 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:41:43.645 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:44.879 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-16T07:41:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:41:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:44.879 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:41:45.126 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-nbd 2024-04-16T07:41:45.726 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:56 ago on Tue 16 Apr 2024 07:36:49 AM UTC. 2024-04-16T07:41:45.885 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-16T07:41:45.885 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-04-16T07:41:45.885 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:45.886 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-16T07:41:45.886 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 175 k 2024-04-16T07:41:45.886 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:45.886 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-16T07:41:45.886 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-16T07:41:45.886 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-04-16T07:41:45.886 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:45.886 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 175 k 2024-04-16T07:41:45.886 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 673 k 2024-04-16T07:41:45.886 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-16T07:41:46.772 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-nbd-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 198 kB/s | 175 kB 00:00 2024-04-16T07:41:46.772 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:46.773 INFO:teuthology.orchestra.run.smithi031.stdout:Total 198 kB/s | 175 kB 00:00 2024-04-16T07:41:46.773 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-16T07:41:46.785 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-16T07:41:46.785 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-16T07:41:46.848 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-16T07:41:46.848 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-16T07:41:47.005 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-16T07:41:47.106 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:47.527 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:47.828 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:47.828 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:47.828 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-16T07:41:47.828 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:41:47.828 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:41:47.828 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-16T07:41:48.078 DEBUG:teuthology.parallel:result is None 2024-04-16T07:41:55.749 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-mirror-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 265 kB/s | 3.4 MB 00:13 2024-04-16T07:41:55.749 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:55.750 INFO:teuthology.orchestra.run.smithi094.stdout:Total 264 kB/s | 3.4 MB 00:13 2024-04-16T07:41:55.750 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-16T07:41:55.763 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-16T07:41:55.763 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-16T07:41:55.842 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-16T07:41:55.843 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-16T07:41:56.408 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-16T07:41:56.558 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:56.582 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:56.583 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:41:56.583 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T07:41:56.583 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:41:56.583 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:41:56.583 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:57.859 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:41:57.859 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:57.859 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-16T07:41:57.859 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:41:57.859 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:57.859 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:41:58.126 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-nbd 2024-04-16T07:41:58.750 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:05:15 ago on Tue 16 Apr 2024 07:36:43 AM UTC. 2024-04-16T07:41:58.906 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-16T07:41:58.906 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-04-16T07:41:58.907 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:58.907 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-16T07:41:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 175 k 2024-04-16T07:41:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:58.907 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-16T07:41:58.907 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-16T07:41:58.907 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-04-16T07:41:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:41:58.907 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 175 k 2024-04-16T07:41:58.907 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 673 k 2024-04-16T07:41:58.907 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-16T07:42:00.962 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-nbd-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 85 kB/s | 175 kB 00:02 2024-04-16T07:42:00.962 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:42:00.963 INFO:teuthology.orchestra.run.smithi094.stdout:Total 85 kB/s | 175 kB 00:02 2024-04-16T07:42:00.963 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-16T07:42:00.975 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-16T07:42:00.975 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-16T07:42:01.036 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-16T07:42:01.036 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-16T07:42:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-16T07:42:01.293 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:42:01.734 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:42:02.025 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:42:02.026 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:02.026 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-16T07:42:02.026 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:42:02.026 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:02.026 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-16T07:42:02.334 DEBUG:teuthology.parallel:result is None 2024-04-16T07:42:02.335 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:42:02.484 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T07:42:02.509 INFO:teuthology.orchestra.run.smithi031.stdout:19.0.0-3109.g30b0aa00.el9 2024-04-16T07:42:02.510 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3109.g30b0aa00.el9 2024-04-16T07:42:02.510 INFO:teuthology.task.install:The correct ceph version 19.0.0-3109.g30b0aa00 is installed. 2024-04-16T07:42:02.512 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:42:02.739 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T07:42:02.761 INFO:teuthology.orchestra.run.smithi094.stdout:19.0.0-3109.g30b0aa00.el9 2024-04-16T07:42:02.761 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3109.g30b0aa00.el9 2024-04-16T07:42:02.761 INFO:teuthology.task.install:The correct ceph version 19.0.0-3109.g30b0aa00 is installed. 2024-04-16T07:42:02.763 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-16T07:42:02.763 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-16T07:42:02.763 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T07:42:02.796 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-16T07:42:02.796 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T07:42:02.831 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-16T07:42:02.832 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-16T07:42:02.832 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T07:42:02.868 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T07:42:02.939 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-16T07:42:02.939 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T07:42:02.967 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T07:42:03.047 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-16T07:42:03.047 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-16T07:42:03.047 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T07:42:03.079 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T07:42:03.150 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-16T07:42:03.150 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T07:42:03.178 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T07:42:03.246 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-16T07:42:03.246 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-16T07:42:03.246 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T07:42:03.274 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T07:42:03.342 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-16T07:42:03.342 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T07:42:03.370 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T07:42:03.439 INFO:teuthology.run_tasks:Running task ceph... 2024-04-16T07:42:03.522 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-16T07:42:03.522 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /var/log/ceph 2024-04-16T07:42:03.524 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 777 /var/log/ceph 2024-04-16T07:42:03.552 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-16T07:42:03.552 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-16T07:42:03.593 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-16T07:42:03.620 INFO:tasks.ceph:Creating extra log directories... 2024-04-16T07:42:03.621 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-16T07:42:03.660 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-16T07:42:03.699 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-16T07:42:03.699 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-16T07:42:03.699 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657957', 'branch': 'wip-matanb-crimson-replicated-april-15', 'description': 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests}', 'email': None, 'first_in_suite': False, 'job_id': '7657957', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-replicated-april-15'}, 'ceph': {'conf': {'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-matanb-crimson-replicated-april-15', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 7594, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-matanb-crimson-replicated-april-15', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'targets': {'smithi031.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBS07dOWcaLjUiOU6r19LBrr0T8Y6Zjb8ZCWZeDak9Zu/sx6QisBlA9o2b3LdPcm1uZCdsJBZZe7jXOMEGadN0c=', 'smithi094.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCB2K1ya0mQ2pOgjC1b+JEgG9txeTVJWCARqu4FLJYKv0kBC4TWbRYstDVBDoII6KaXMqEibABpKWxNkhPZ2Des='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'radosbench': {'clients': ['client.0'], 'time': 150}}, {'admin_socket': {'client.0': {'objecter_requests': {'test': 'http://git.ceph.com/?p={repo};a=blob_plain;f=src/test/admin_socket/objecter_requests;hb={branch}'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-16_07:22:04', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-16T07:42:03.700 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-16T07:42:03.730 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-16T07:42:03.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T07:42:03.785 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T07:42:03.814 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-16T07:42:03.814 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T07:42:03.867 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T07:42:03.868 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2024-04-16T07:42:03.923 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T07:42:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:42:03.924 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-16T07:42:03.924 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:42:03.924 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:42:03.924 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-16 07:41:43.899170468 +0000 2024-04-16T07:42:03.924 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-16 07:35:53.717093749 +0000 2024-04-16T07:42:03.924 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-16 07:35:53.717093749 +0000 2024-04-16T07:42:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-04-16 07:35:53.717093749 +0000 2024-04-16T07:42:03.925 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T07:42:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-16T07:42:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-16T07:42:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.00025745 s, 2.0 MB/s 2024-04-16T07:42:03.991 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T07:42:04.047 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2024-04-16T07:42:04.103 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T07:42:04.103 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:42:04.103 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-16T07:42:04.103 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:42:04.103 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:42:04.103 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-16 07:41:43.900170452 +0000 2024-04-16T07:42:04.104 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-16 07:35:54.078087838 +0000 2024-04-16T07:42:04.104 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-16 07:35:54.078087838 +0000 2024-04-16T07:42:04.104 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-04-16 07:35:54.078087838 +0000 2024-04-16T07:42:04.104 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T07:42:04.169 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-16T07:42:04.170 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-16T07:42:04.170 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000201236 s, 2.5 MB/s 2024-04-16T07:42:04.171 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T07:42:04.226 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2024-04-16T07:42:04.282 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T07:42:04.282 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:42:04.282 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-16T07:42:04.282 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:42:04.282 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:42:04.282 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-16 07:41:43.900170452 +0000 2024-04-16T07:42:04.282 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-16 07:35:54.434082009 +0000 2024-04-16T07:42:04.282 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-16 07:35:54.434082009 +0000 2024-04-16T07:42:04.282 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-04-16 07:35:54.434082009 +0000 2024-04-16T07:42:04.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T07:42:04.349 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-16T07:42:04.349 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-16T07:42:04.349 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000251189 s, 2.0 MB/s 2024-04-16T07:42:04.350 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T07:42:04.406 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2024-04-16T07:42:04.461 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T07:42:04.461 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:42:04.461 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-16T07:42:04.461 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:42:04.461 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:42:04.461 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-16 07:41:43.900170452 +0000 2024-04-16T07:42:04.461 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-16 07:35:54.785076262 +0000 2024-04-16T07:42:04.462 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-16 07:35:54.785076262 +0000 2024-04-16T07:42:04.462 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-04-16 07:35:54.785076262 +0000 2024-04-16T07:42:04.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T07:42:04.527 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-16T07:42:04.527 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-16T07:42:04.528 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000185737 s, 2.8 MB/s 2024-04-16T07:42:04.529 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T07:42:04.584 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-16T07:42:04.585 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-16T07:42:04.585 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T07:42:04.601 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T07:42:04.602 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_1 2024-04-16T07:42:04.659 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T07:42:04.659 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:42:04.659 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-16T07:42:04.659 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:42:04.659 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:42:04.659 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-04-16 07:41:56.833604528 +0000 2024-04-16T07:42:04.659 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-04-16 07:35:49.266783895 +0000 2024-04-16T07:42:04.659 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-04-16 07:35:49.266783895 +0000 2024-04-16T07:42:04.659 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-04-16 07:35:49.266783895 +0000 2024-04-16T07:42:04.660 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T07:42:04.727 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-04-16T07:42:04.727 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-04-16T07:42:04.727 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000111003 s, 4.6 MB/s 2024-04-16T07:42:04.729 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T07:42:04.786 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_2 2024-04-16T07:42:04.842 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T07:42:04.842 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:42:04.842 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-16T07:42:04.842 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:42:04.842 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:42:04.842 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-04-16 07:41:56.834604511 +0000 2024-04-16T07:42:04.842 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-04-16 07:35:49.617777834 +0000 2024-04-16T07:42:04.842 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-04-16 07:35:49.617777834 +0000 2024-04-16T07:42:04.842 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-04-16 07:35:49.617777834 +0000 2024-04-16T07:42:04.843 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T07:42:04.909 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-04-16T07:42:04.909 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-04-16T07:42:04.909 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000100483 s, 5.1 MB/s 2024-04-16T07:42:04.911 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T07:42:04.967 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_3 2024-04-16T07:42:05.023 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T07:42:05.023 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:42:05.023 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-16T07:42:05.024 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:42:05.024 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:42:05.024 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-04-16 07:41:56.834604511 +0000 2024-04-16T07:42:05.024 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-04-16 07:35:49.953772033 +0000 2024-04-16T07:42:05.024 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-04-16 07:35:49.953772033 +0000 2024-04-16T07:42:05.024 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-04-16 07:35:49.953772033 +0000 2024-04-16T07:42:05.024 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T07:42:05.089 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-04-16T07:42:05.089 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-04-16T07:42:05.089 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 7.6557e-05 s, 6.7 MB/s 2024-04-16T07:42:05.090 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T07:42:05.146 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_4 2024-04-16T07:42:05.201 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T07:42:05.202 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:42:05.202 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-16T07:42:05.202 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:42:05.202 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:42:05.202 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-04-16 07:41:56.835604493 +0000 2024-04-16T07:42:05.202 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-04-16 07:35:50.290766214 +0000 2024-04-16T07:42:05.202 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-04-16 07:35:50.290766214 +0000 2024-04-16T07:42:05.202 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-04-16 07:35:50.290766214 +0000 2024-04-16T07:42:05.202 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T07:42:05.269 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-04-16T07:42:05.269 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-04-16T07:42:05.269 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000100603 s, 5.1 MB/s 2024-04-16T07:42:05.270 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T07:42:05.326 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-16T07:42:05.326 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-04-16T07:42:05.326 INFO:tasks.ceph:Generating config... 2024-04-16T07:42:05.328 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-16T07:42:05.328 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-16T07:42:05.328 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-16T07:42:05.328 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-16T07:42:05.328 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-16T07:42:05.328 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-04-16T07:42:05.328 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-16T07:42:05.328 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-16T07:42:05.328 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-16T07:42:05.328 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-16T07:42:05.328 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-16T07:42:05.328 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-16T07:42:05.329 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-16T07:42:05.329 INFO:tasks.ceph:[osd] err to stderr = True 2024-04-16T07:42:05.329 INFO:tasks.ceph:[osd] log flush on exit = True 2024-04-16T07:42:05.329 INFO:tasks.ceph:[osd] log to file = False 2024-04-16T07:42:05.329 INFO:tasks.ceph:[osd] log to stderr = True 2024-04-16T07:42:05.329 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-16T07:42:05.329 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-16T07:42:05.329 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-16T07:42:05.329 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-16T07:42:05.329 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-04-16T07:42:05.329 INFO:tasks.ceph:[osd] osd delete sleep = 1 2024-04-16T07:42:05.329 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-16T07:42:05.329 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-04-16T07:42:05.329 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-16T07:42:05.330 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-16T07:42:05.330 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-04-16T07:42:05.330 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-04-16T07:42:05.330 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-16T07:42:05.330 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-04-16T07:42:05.330 INFO:tasks.ceph:[client.0] admin socket = /var/run/ceph/ceph-$name.asok 2024-04-16T07:42:05.330 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-16T07:42:05.330 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-16T07:42:05.330 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-16T07:42:05.330 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-16T07:42:05.330 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-16T07:42:05.330 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-16T07:42:05.330 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-16T07:42:05.330 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-04-16T07:42:05.330 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2024-04-16T07:42:05.331 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-16T07:42:05.331 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-16T07:42:05.331 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-16T07:42:05.331 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-16T07:42:05.331 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-16T07:42:05.331 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-04-16T07:42:05.331 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-16T07:42:05.331 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-16T07:42:05.331 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-16T07:42:05.331 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-16T07:42:05.331 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-04-16T07:42:05.331 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-04-16T07:42:05.331 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-04-16T07:42:05.331 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-04-16T07:42:05.331 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-04-16T07:42:05.332 INFO:tasks.ceph:Setting up mon.a... 2024-04-16T07:42:05.332 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-16T07:42:05.395 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.keyring 2024-04-16T07:42:05.398 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-16T07:42:05.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T07:42:05.482 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.31')] 2024-04-16T07:42:05.482 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.31', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-04-16T07:42:05.483 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-16T07:42:05.483 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T07:42:05.537 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.31 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:42:05.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:05.627+0000 7fad6fa2ff00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:05.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:05.627+0000 7fad6fa2ff00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:05.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:05.627+0000 7fad6fa2ff00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:05.628 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:42:05.629 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: generated fsid e7bebb51-b79a-49a6-ad6e-f22a47077039 2024-04-16T07:42:05.629 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = quincy 2024-04-16T07:42:05.629 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 0 2024-04-16T07:42:05.629 INFO:teuthology.orchestra.run.smithi031.stdout:fsid e7bebb51-b79a-49a6-ad6e-f22a47077039 2024-04-16T07:42:05.629 INFO:teuthology.orchestra.run.smithi031.stdout:last_changed 2024-04-16T07:42:05.629759+0000 2024-04-16T07:42:05.629 INFO:teuthology.orchestra.run.smithi031.stdout:created 2024-04-16T07:42:05.629759+0000 2024-04-16T07:42:05.629 INFO:teuthology.orchestra.run.smithi031.stdout:min_mon_release 17 (quincy) 2024-04-16T07:42:05.629 INFO:teuthology.orchestra.run.smithi031.stdout:election_strategy: 1 2024-04-16T07:42:05.629 INFO:teuthology.orchestra.run.smithi031.stdout:0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-16T07:42:05.629 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-16T07:42:05.630 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T07:42:05.645 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e7bebb51-b79a-49a6-ad6e-f22a47077039... 2024-04-16T07:42:05.646 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-16T07:42:05.688 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-16T07:42:05.738 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2024-04-16T07:42:05.738 INFO:teuthology.orchestra.run.smithi031.stdout: chdir = "" 2024-04-16T07:42:05.738 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T07:42:05.738 INFO:teuthology.orchestra.run.smithi031.stdout: auth supported = cephx 2024-04-16T07:42:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: filestore xattr use omap = true 2024-04-16T07:42:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: mon clock drift allowed = 1.000 2024-04-16T07:42:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: osd crush chooseleaf type = 0 2024-04-16T07:42:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: auth debug = true 2024-04-16T07:42:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on old message = true 2024-04-16T07:42:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on bug = true 2024-04-16T07:42:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T07:42:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: mon pg warn max object skew = 0 2024-04-16T07:42:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T07:42:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T07:42:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default size = 2 2024-04-16T07:42:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow primary affinity = true 2024-04-16T07:42:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow pg remap = true 2024-04-16T07:42:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on legacy crush tunables = false 2024-04-16T07:42:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on crush straw calc version zero = false 2024-04-16T07:42:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on no sortbitwise = false 2024-04-16T07:42:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on osd down out interval zero = false 2024-04-16T07:42:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on too few osds = false 2024-04-16T07:42:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T07:42:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T07:42:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: mon_allow_pool_size_one = true 2024-04-16T07:42:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T07:42:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: osd default data pool replay window = 5 2024-04-16T07:42:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: mon allow pool delete = true 2024-04-16T07:42:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: mon cluster log file level = debug 2024-04-16T07:42:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: debug asserts on shutdown = true 2024-04-16T07:42:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: mon health detail to clog = false 2024-04-16T07:42:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: mon host = 172.21.15.31 2024-04-16T07:42:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-16T07:42:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: ms client mode = crc 2024-04-16T07:42:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: ms cluster mode = crc 2024-04-16T07:42:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: ms mon client mode = crc 2024-04-16T07:42:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: ms mon cluster mode = crc 2024-04-16T07:42:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: ms mon service mode = crc 2024-04-16T07:42:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: ms service mode = crc 2024-04-16T07:42:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_min_size = 2 2024-04-16T07:42:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_size = 3 2024-04-16T07:42:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = e7bebb51-b79a-49a6-ad6e-f22a47077039 2024-04-16T07:42:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.742 INFO:teuthology.orchestra.run.smithi031.stdout:[osd] 2024-04-16T07:42:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: osd journal size = 100 2024-04-16T07:42:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub load threshold = 5.0 2024-04-16T07:42:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub max interval = 120 2024-04-16T07:42:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock profile = high_recovery_ops 2024-04-16T07:42:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: osd recover clone overlap = true 2024-04-16T07:42:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: osd recovery max chunk = 1048576 2024-04-16T07:42:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug shutdown = true 2024-04-16T07:42:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug op order = true 2024-04-16T07:42:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify stray on activate = true 2024-04-16T07:42:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug trim objects = true 2024-04-16T07:42:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: osd open classes on start = true 2024-04-16T07:42:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug pg log writeout = true 2024-04-16T07:42:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: osd deep scrub update digest min age = 30 2024-04-16T07:42:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: osd map max advance = 10 2024-04-16T07:42:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: journal zero on create = true 2024-04-16T07:42:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: filestore ondisk finisher threads = 3 2024-04-16T07:42:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: filestore apply finisher threads = 3 2024-04-16T07:42:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: bdev debug aio = true 2024-04-16T07:42:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug misdirected ops = true 2024-04-16T07:42:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: debug monc = 20 2024-04-16T07:42:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore block size = 96636764160 2024-04-16T07:42:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore compression algorithm = snappy 2024-04-16T07:42:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore compression mode = aggressive 2024-04-16T07:42:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore fsck on mount = True 2024-04-16T07:42:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore rocksdb cf = False 2024-04-16T07:42:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: crimson alien thread cpu cores = 6-7 2024-04-16T07:42:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: crimson osd obc lru size = 10 2024-04-16T07:42:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluefs = 20 2024-04-16T07:42:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluestore = 20 2024-04-16T07:42:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-04-16T07:42:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 20 2024-04-16T07:42:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: debug rocksdb = 10 2024-04-16T07:42:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: err to stderr = True 2024-04-16T07:42:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: log flush on exit = True 2024-04-16T07:42:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: log to file = False 2024-04-16T07:42:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: log to stderr = True 2024-04-16T07:42:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd backfillfull_ratio = 0.85 2024-04-16T07:42:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd full ratio = 0.9 2024-04-16T07:42:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd nearfull ratio = 0.8 2024-04-16T07:42:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: osd blocked scrub grace period = 3600 2024-04-16T07:42:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug reject backfill probability = 0.3 2024-04-16T07:42:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: osd delete sleep = 1 2024-04-16T07:42:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: osd failsafe full ratio = 0.95 2024-04-16T07:42:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: osd max backfills = 3 2024-04-16T07:42:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: osd max markdown count = 1000 2024-04-16T07:42:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: osd objectstore = bluestore 2024-04-16T07:42:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub min interval = 60 2024-04-16T07:42:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: osd shutdown pgref assert = True 2024-04-16T07:42:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: osd snap trim sleep = 2 2024-04-16T07:42:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.745 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr] 2024-04-16T07:42:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-04-16T07:42:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 20 2024-04-16T07:42:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2024-04-16T07:42:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2024-04-16T07:42:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2024-04-16T07:42:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2024-04-16T07:42:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: mgr/telemetry/nag = false 2024-04-16T07:42:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.746 INFO:teuthology.orchestra.run.smithi031.stdout:[mon] 2024-04-16T07:42:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-04-16T07:42:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2024-04-16T07:42:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 20 2024-04-16T07:42:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2024-04-16T07:42:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: mon data avail warn = 5 2024-04-16T07:42:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: mon mgr mkfs grace = 240 2024-04-16T07:42:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2024-04-16T07:42:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd reporter subtree level = osd 2024-04-16T07:42:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd prime pg temp = true 2024-04-16T07:42:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2024-04-16T07:42:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T07:42:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T07:42:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T07:42:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T07:42:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T07:42:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T07:42:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: # 1m isn't quite enough 2024-04-16T07:42:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: mon_down_mkfs_grace = 2m 2024-04-16T07:42:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_filestore_osds = false 2024-04-16T07:42:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: mon min osdmap epochs = 50 2024-04-16T07:42:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: mon osdmap full prune interval = 2 2024-04-16T07:42:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: mon osdmap full prune min = 15 2024-04-16T07:42:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: mon osdmap full prune txsize = 2 2024-04-16T07:42:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default crimson = True 2024-04-16T07:42:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: paxos service trim min = 10 2024-04-16T07:42:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:05.748 INFO:teuthology.orchestra.run.smithi031.stdout:[client] 2024-04-16T07:42:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: rgw cache enabled = true 2024-04-16T07:42:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable ops log = true 2024-04-16T07:42:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable usage log = true 2024-04-16T07:42:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T07:42:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T07:42:05.748 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.a] 2024-04-16T07:42:05.748 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-04-16T07:42:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/ceph-$name.asok 2024-04-16T07:42:05.748 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.0] 2024-04-16T07:42:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: crimson seastar cpu cores = 0-2 2024-04-16T07:42:05.749 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.1] 2024-04-16T07:42:05.749 INFO:teuthology.orchestra.run.smithi031.stdout: crimson seastar cpu cores = 3-5 2024-04-16T07:42:05.749 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.2] 2024-04-16T07:42:05.749 INFO:teuthology.orchestra.run.smithi031.stdout: crimson seastar cpu cores = 0-2 2024-04-16T07:42:05.749 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.3] 2024-04-16T07:42:05.749 INFO:teuthology.orchestra.run.smithi031.stdout: crimson seastar cpu cores = 3-5 2024-04-16T07:42:05.749 INFO:teuthology.orchestra.run.smithi094.stdout:[global] 2024-04-16T07:42:05.749 INFO:teuthology.orchestra.run.smithi094.stdout: chdir = "" 2024-04-16T07:42:05.749 INFO:teuthology.orchestra.run.smithi094.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T07:42:05.749 INFO:teuthology.orchestra.run.smithi094.stdout: auth supported = cephx 2024-04-16T07:42:05.749 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.750 INFO:teuthology.orchestra.run.smithi094.stdout: filestore xattr use omap = true 2024-04-16T07:42:05.750 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.750 INFO:teuthology.orchestra.run.smithi094.stdout: mon clock drift allowed = 1.000 2024-04-16T07:42:05.750 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.750 INFO:teuthology.orchestra.run.smithi094.stdout: osd crush chooseleaf type = 0 2024-04-16T07:42:05.750 INFO:teuthology.orchestra.run.smithi094.stdout: auth debug = true 2024-04-16T07:42:05.750 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.750 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on old message = true 2024-04-16T07:42:05.750 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on bug = true 2024-04-16T07:42:05.750 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.750 INFO:teuthology.orchestra.run.smithi094.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T07:42:05.750 INFO:teuthology.orchestra.run.smithi094.stdout: mon pg warn max object skew = 0 2024-04-16T07:42:05.750 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.750 INFO:teuthology.orchestra.run.smithi094.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T07:42:05.750 INFO:teuthology.orchestra.run.smithi094.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T07:42:05.750 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.751 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default size = 2 2024-04-16T07:42:05.751 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.751 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow primary affinity = true 2024-04-16T07:42:05.751 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow pg remap = true 2024-04-16T07:42:05.751 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on legacy crush tunables = false 2024-04-16T07:42:05.751 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on crush straw calc version zero = false 2024-04-16T07:42:05.751 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on no sortbitwise = false 2024-04-16T07:42:05.751 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on osd down out interval zero = false 2024-04-16T07:42:05.751 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on too few osds = false 2024-04-16T07:42:05.751 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T07:42:05.751 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T07:42:05.751 INFO:teuthology.orchestra.run.smithi094.stdout: mon_allow_pool_size_one = true 2024-04-16T07:42:05.751 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.751 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T07:42:05.751 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.752 INFO:teuthology.orchestra.run.smithi094.stdout: osd default data pool replay window = 5 2024-04-16T07:42:05.752 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.752 INFO:teuthology.orchestra.run.smithi094.stdout: mon allow pool delete = true 2024-04-16T07:42:05.752 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.752 INFO:teuthology.orchestra.run.smithi094.stdout: mon cluster log file level = debug 2024-04-16T07:42:05.752 INFO:teuthology.orchestra.run.smithi094.stdout: debug asserts on shutdown = true 2024-04-16T07:42:05.752 INFO:teuthology.orchestra.run.smithi094.stdout: mon health detail to clog = false 2024-04-16T07:42:05.752 INFO:teuthology.orchestra.run.smithi094.stdout: mon host = 172.21.15.31 2024-04-16T07:42:05.752 INFO:teuthology.orchestra.run.smithi094.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-16T07:42:05.752 INFO:teuthology.orchestra.run.smithi094.stdout: ms client mode = crc 2024-04-16T07:42:05.752 INFO:teuthology.orchestra.run.smithi094.stdout: ms cluster mode = crc 2024-04-16T07:42:05.752 INFO:teuthology.orchestra.run.smithi094.stdout: ms mon client mode = crc 2024-04-16T07:42:05.752 INFO:teuthology.orchestra.run.smithi094.stdout: ms mon cluster mode = crc 2024-04-16T07:42:05.752 INFO:teuthology.orchestra.run.smithi094.stdout: ms mon service mode = crc 2024-04-16T07:42:05.752 INFO:teuthology.orchestra.run.smithi094.stdout: ms service mode = crc 2024-04-16T07:42:05.752 INFO:teuthology.orchestra.run.smithi094.stdout: osd_pool_default_min_size = 2 2024-04-16T07:42:05.752 INFO:teuthology.orchestra.run.smithi094.stdout: osd_pool_default_size = 3 2024-04-16T07:42:05.753 INFO:teuthology.orchestra.run.smithi094.stdout: fsid = e7bebb51-b79a-49a6-ad6e-f22a47077039 2024-04-16T07:42:05.753 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.753 INFO:teuthology.orchestra.run.smithi094.stdout:[osd] 2024-04-16T07:42:05.753 INFO:teuthology.orchestra.run.smithi094.stdout: osd journal size = 100 2024-04-16T07:42:05.753 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.753 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub load threshold = 5.0 2024-04-16T07:42:05.753 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub max interval = 120 2024-04-16T07:42:05.753 INFO:teuthology.orchestra.run.smithi094.stdout: osd mclock profile = high_recovery_ops 2024-04-16T07:42:05.753 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.753 INFO:teuthology.orchestra.run.smithi094.stdout: osd recover clone overlap = true 2024-04-16T07:42:05.753 INFO:teuthology.orchestra.run.smithi094.stdout: osd recovery max chunk = 1048576 2024-04-16T07:42:05.753 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.753 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug shutdown = true 2024-04-16T07:42:05.753 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug op order = true 2024-04-16T07:42:05.753 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify stray on activate = true 2024-04-16T07:42:05.753 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug trim objects = true 2024-04-16T07:42:05.753 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.754 INFO:teuthology.orchestra.run.smithi094.stdout: osd open classes on start = true 2024-04-16T07:42:05.754 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug pg log writeout = true 2024-04-16T07:42:05.754 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.754 INFO:teuthology.orchestra.run.smithi094.stdout: osd deep scrub update digest min age = 30 2024-04-16T07:42:05.754 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.754 INFO:teuthology.orchestra.run.smithi094.stdout: osd map max advance = 10 2024-04-16T07:42:05.754 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.754 INFO:teuthology.orchestra.run.smithi094.stdout: journal zero on create = true 2024-04-16T07:42:05.754 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.754 INFO:teuthology.orchestra.run.smithi094.stdout: filestore ondisk finisher threads = 3 2024-04-16T07:42:05.754 INFO:teuthology.orchestra.run.smithi094.stdout: filestore apply finisher threads = 3 2024-04-16T07:42:05.754 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.754 INFO:teuthology.orchestra.run.smithi094.stdout: bdev debug aio = true 2024-04-16T07:42:05.754 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug misdirected ops = true 2024-04-16T07:42:05.755 INFO:teuthology.orchestra.run.smithi094.stdout: debug monc = 20 2024-04-16T07:42:05.755 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore block size = 96636764160 2024-04-16T07:42:05.755 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore compression algorithm = snappy 2024-04-16T07:42:05.755 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore compression mode = aggressive 2024-04-16T07:42:05.755 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore fsck on mount = True 2024-04-16T07:42:05.755 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore rocksdb cf = False 2024-04-16T07:42:05.755 INFO:teuthology.orchestra.run.smithi094.stdout: crimson alien thread cpu cores = 6-7 2024-04-16T07:42:05.755 INFO:teuthology.orchestra.run.smithi094.stdout: crimson osd obc lru size = 10 2024-04-16T07:42:05.755 INFO:teuthology.orchestra.run.smithi094.stdout: debug bluefs = 20 2024-04-16T07:42:05.755 INFO:teuthology.orchestra.run.smithi094.stdout: debug bluestore = 20 2024-04-16T07:42:05.755 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-04-16T07:42:05.755 INFO:teuthology.orchestra.run.smithi094.stdout: debug osd = 20 2024-04-16T07:42:05.755 INFO:teuthology.orchestra.run.smithi094.stdout: debug rocksdb = 10 2024-04-16T07:42:05.755 INFO:teuthology.orchestra.run.smithi094.stdout: err to stderr = True 2024-04-16T07:42:05.755 INFO:teuthology.orchestra.run.smithi094.stdout: log flush on exit = True 2024-04-16T07:42:05.756 INFO:teuthology.orchestra.run.smithi094.stdout: log to file = False 2024-04-16T07:42:05.756 INFO:teuthology.orchestra.run.smithi094.stdout: log to stderr = True 2024-04-16T07:42:05.756 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd backfillfull_ratio = 0.85 2024-04-16T07:42:05.756 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd full ratio = 0.9 2024-04-16T07:42:05.756 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd nearfull ratio = 0.8 2024-04-16T07:42:05.756 INFO:teuthology.orchestra.run.smithi094.stdout: osd blocked scrub grace period = 3600 2024-04-16T07:42:05.756 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug reject backfill probability = 0.3 2024-04-16T07:42:05.756 INFO:teuthology.orchestra.run.smithi094.stdout: osd delete sleep = 1 2024-04-16T07:42:05.756 INFO:teuthology.orchestra.run.smithi094.stdout: osd failsafe full ratio = 0.95 2024-04-16T07:42:05.756 INFO:teuthology.orchestra.run.smithi094.stdout: osd max backfills = 3 2024-04-16T07:42:05.756 INFO:teuthology.orchestra.run.smithi094.stdout: osd max markdown count = 1000 2024-04-16T07:42:05.756 INFO:teuthology.orchestra.run.smithi094.stdout: osd objectstore = bluestore 2024-04-16T07:42:05.756 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub min interval = 60 2024-04-16T07:42:05.756 INFO:teuthology.orchestra.run.smithi094.stdout: osd shutdown pgref assert = True 2024-04-16T07:42:05.756 INFO:teuthology.orchestra.run.smithi094.stdout: osd snap trim sleep = 2 2024-04-16T07:42:05.756 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.757 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr] 2024-04-16T07:42:05.757 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-04-16T07:42:05.757 INFO:teuthology.orchestra.run.smithi094.stdout: debug mgr = 20 2024-04-16T07:42:05.757 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2024-04-16T07:42:05.757 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2024-04-16T07:42:05.757 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2024-04-16T07:42:05.757 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2024-04-16T07:42:05.757 INFO:teuthology.orchestra.run.smithi094.stdout: mgr/telemetry/nag = false 2024-04-16T07:42:05.757 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.757 INFO:teuthology.orchestra.run.smithi094.stdout:[mon] 2024-04-16T07:42:05.757 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-04-16T07:42:05.757 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2024-04-16T07:42:05.757 INFO:teuthology.orchestra.run.smithi094.stdout: debug paxos = 20 2024-04-16T07:42:05.757 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2024-04-16T07:42:05.757 INFO:teuthology.orchestra.run.smithi094.stdout: mon data avail warn = 5 2024-04-16T07:42:05.757 INFO:teuthology.orchestra.run.smithi094.stdout: mon mgr mkfs grace = 240 2024-04-16T07:42:05.757 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2024-04-16T07:42:05.758 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd reporter subtree level = osd 2024-04-16T07:42:05.758 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd prime pg temp = true 2024-04-16T07:42:05.758 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2024-04-16T07:42:05.758 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.758 INFO:teuthology.orchestra.run.smithi094.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T07:42:05.758 INFO:teuthology.orchestra.run.smithi094.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T07:42:05.758 INFO:teuthology.orchestra.run.smithi094.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T07:42:05.758 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.758 INFO:teuthology.orchestra.run.smithi094.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T07:42:05.758 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T07:42:05.758 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T07:42:05.758 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.758 INFO:teuthology.orchestra.run.smithi094.stdout: # 1m isn't quite enough 2024-04-16T07:42:05.758 INFO:teuthology.orchestra.run.smithi094.stdout: mon_down_mkfs_grace = 2m 2024-04-16T07:42:05.758 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.758 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_filestore_osds = false 2024-04-16T07:42:05.758 INFO:teuthology.orchestra.run.smithi094.stdout: mon min osdmap epochs = 50 2024-04-16T07:42:05.759 INFO:teuthology.orchestra.run.smithi094.stdout: mon osdmap full prune interval = 2 2024-04-16T07:42:05.759 INFO:teuthology.orchestra.run.smithi094.stdout: mon osdmap full prune min = 15 2024-04-16T07:42:05.759 INFO:teuthology.orchestra.run.smithi094.stdout: mon osdmap full prune txsize = 2 2024-04-16T07:42:05.759 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default crimson = True 2024-04-16T07:42:05.759 INFO:teuthology.orchestra.run.smithi094.stdout: paxos service trim min = 10 2024-04-16T07:42:05.759 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-16T07:42:05.759 INFO:teuthology.orchestra.run.smithi094.stdout:[client] 2024-04-16T07:42:05.759 INFO:teuthology.orchestra.run.smithi094.stdout: rgw cache enabled = true 2024-04-16T07:42:05.759 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable ops log = true 2024-04-16T07:42:05.759 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable usage log = true 2024-04-16T07:42:05.759 INFO:teuthology.orchestra.run.smithi094.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T07:42:05.759 INFO:teuthology.orchestra.run.smithi094.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T07:42:05.759 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.a] 2024-04-16T07:42:05.759 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-04-16T07:42:05.759 INFO:teuthology.orchestra.run.smithi094.stdout: admin socket = /var/run/ceph/ceph-$name.asok 2024-04-16T07:42:05.760 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.0] 2024-04-16T07:42:05.760 INFO:teuthology.orchestra.run.smithi094.stdout: crimson seastar cpu cores = 0-2 2024-04-16T07:42:05.760 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.1] 2024-04-16T07:42:05.760 INFO:teuthology.orchestra.run.smithi094.stdout: crimson seastar cpu cores = 3-5 2024-04-16T07:42:05.760 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.2] 2024-04-16T07:42:05.760 INFO:teuthology.orchestra.run.smithi094.stdout: crimson seastar cpu cores = 0-2 2024-04-16T07:42:05.760 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.3] 2024-04-16T07:42:05.760 INFO:teuthology.orchestra.run.smithi094.stdout: crimson seastar cpu cores = 3-5 2024-04-16T07:42:05.761 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-16T07:42:05.761 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-16T07:42:05.828 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-16T07:42:05.828 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-16T07:42:05.828 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-16T07:42:05.843 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-16T07:42:05.843 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-16T07:42:05.898 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi031.front.sepia.ceph.com 2024-04-16T07:42:05.898 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-16T07:42:05.898 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-16T07:42:05.898 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T07:42:05.979 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-16T07:42:05.979 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:42:06.034 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi094.front.sepia.ceph.com 2024-04-16T07:42:06.034 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-16T07:42:06.034 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-16T07:42:06.034 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T07:42:06.076 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-16T07:42:06.076 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:42:06.131 INFO:tasks.ceph:Setting up mon nodes... 2024-04-16T07:42:06.132 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-16T07:42:06.132 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-16T07:42:06.278 INFO:teuthology.orchestra.run.smithi094.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-16T07:42:06.281 INFO:tasks.ceph:Setting up mds nodes... 2024-04-16T07:42:06.282 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-16T07:42:06.282 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-16T07:42:06.334 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-16T07:42:06.351 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-16T07:42:06.404 INFO:teuthology.orchestra.run.smithi094.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-16T07:42:06.420 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-16T07:42:06.420 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-04-16T07:42:06.420 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-16T07:42:06.448 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-16T07:42:06.448 INFO:tasks.ceph:role: osd.0 2024-04-16T07:42:06.448 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com 2024-04-16T07:42:06.448 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-16T07:42:06.523 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T07:42:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T07:42:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T07:42:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T07:42:06.524 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T07:42:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-04-16T07:42:06.524 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T07:42:06.524 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T07:42:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T07:42:06.524 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T07:42:07.039 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-04-16T07:42:07.041 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-04-16T07:42:07.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-16T07:42:07.185 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-16T07:42:07.219 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-16T07:42:07.288 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-16T07:42:07.288 INFO:tasks.ceph:role: osd.1 2024-04-16T07:42:07.288 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com 2024-04-16T07:42:07.288 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-16T07:42:07.364 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T07:42:07.364 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T07:42:07.364 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T07:42:07.364 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T07:42:07.364 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T07:42:07.364 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-04-16T07:42:07.364 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T07:42:07.364 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T07:42:07.364 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T07:42:07.365 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T07:42:07.821 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-04-16T07:42:07.823 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-04-16T07:42:07.823 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-16T07:42:07.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-16T07:42:07.945 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:42:08.259 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:42:08.260 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2024-04-16 07:42:08,261 seastar - Reactor backend: linux-aio 2024-04-16T07:42:08.271 INFO:teuthology.orchestra.run.smithi031.stderr:==29224==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:42:08.404 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2024-04-16 07:42:08,406 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T07:42:08.404 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2024-04-16 07:42:08,406 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-04-16T07:42:08.405 INFO:teuthology.orchestra.run.smithi031.stderr:DEBUG 2024-04-16 07:42:08,407 [shard 0:main] seastar - reactor::drain 2024-04-16T07:42:08.964 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2024-04-16T07:42:08.965 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2024-04-16T07:42:08.965 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2024-04-16T07:42:08.965 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2024-04-16T07:42:08.965 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2024-04-16T07:42:08.965 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-04-16T07:42:09.024 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:42:09.028 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2024-04-16 07:42:09,029 seastar - Reactor backend: linux-aio 2024-04-16T07:42:09.037 INFO:teuthology.orchestra.run.smithi031.stderr:==29222==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:42:09.102 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb99d759 0xb870220 0xb87179c 0xb8719b2 0xb871b08 0xb871fd1 0x54daf 0x37686 0x5d8ca65 0x5bd1519 0x5bec5dd 0x6ec6e84 0x6ed8657 0x6ed9af4 0x6ed9b79 0x6ed9c7b 0x6ed9d73 0x6ed9f80 0x6eda166 0x6eda4b5 0x6eda859 0x6edaa89 0x6edacf3 0x6edafc0 0x6edb1de 0x6edb296 0x6edb35c 0x6edb43d 0x5a39e18 0x5a39f22 0x5a39fde 0x5a3a09a 0x5a3a156 0x5a3a2a3 0x5a3a42e 0x5a3a7ef 0x5a56730 0xb882ea5 0xb89d13f 0xb93e13d 0xb93f78b 0xb61bf71 0xb61c8ed 0x367a090 0x3feaf 0x3ff5f 0x3463444 2024-04-16T07:42:09.102 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2024-04-16T07:42:13.706 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2024-04-16T07:42:13.706 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2024-04-16T07:42:13.706 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2024-04-16T07:42:13.706 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2024-04-16T07:42:13.706 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2024-04-16T07:42:13.706 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-04-16T07:42:13.707 INFO:teuthology.orchestra.run.smithi031.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid e7bebb51-b79a-49a6-ad6e-f22a47077039 2024-04-16T07:42:13.729 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T07:42:13.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:42:14.061 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:42:14.062 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2024-04-16 07:42:14,063 seastar - Reactor backend: linux-aio 2024-04-16T07:42:14.067 INFO:teuthology.orchestra.run.smithi031.stderr:==29406==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:42:14.175 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb99d759 0xb870220 0xb87179c 0xb8719b2 0xb871b08 0xb871fd1 0x54daf 0xef7ec 0x4a0a5 0x35b7c 0x3689e 0x36900ee 0x5db0fb3 0x5db1404 0x5db17e6 0x5e0d774 0x5d8c2f0 0x5d8c4ca 0x5d8ca56 0x5bd1519 0x5bec5dd 0x6ec6e84 0x6ed8657 0x6ed9af4 0x6ed9b79 0x6ed9c7b 0x6ed9d73 0x6ed9f80 0x6eda166 0x6eda4b5 0x6eda859 0x6edaa89 0x6edacf3 0x6edafc0 0x6edb1de 0x6edb296 0x6edb35c 0x6edb43d 0x5a39e18 0x5a39f22 0x5a39fde 0x5a3a09a 0x5a3a156 0x5a3a2a3 0x5a3a42e 0x5a3a7ef 0x5a56730 0xb882ea5 0xb89d13f 0xb93e13d 0xb93f78b 0xb61bf71 0xb61c8ed 0x37d968a 0x37da1e3 0x3679880 0x3feaf 0x3ff5f 0x3463444 2024-04-16T07:42:14.175 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2024-04-16T07:42:14.201 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2024-04-16 07:42:14,202 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T07:42:14.201 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2024-04-16 07:42:14,202 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-04-16T07:42:14.202 INFO:teuthology.orchestra.run.smithi031.stderr:DEBUG 2024-04-16 07:42:14,203 [shard 0:main] seastar - reactor::drain 2024-04-16T07:42:14.774 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2024-04-16T07:42:14.774 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2024-04-16T07:42:14.775 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2024-04-16T07:42:14.775 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2024-04-16T07:42:14.775 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2024-04-16T07:42:14.775 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-04-16T07:42:14.828 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:42:14.832 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2024-04-16 07:42:14,833 seastar - Reactor backend: linux-aio 2024-04-16T07:42:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:==29404==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:42:14.905 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb99d759 0xb870220 0xb87179c 0xb8719b2 0xb871b08 0xb871fd1 0x54daf 0xec65f 0xdfe22 0x3491a39 0x3491c42 0x34c0ad9 0x36adbf4 0x3fc8445 0x5d8c2ce 0x5d8c824 0x5bd1519 0x5bec5dd 0x6ec6e84 0x6ed8657 0x6ed9af4 0x6ed9b79 0x6ed9c7b 0x6ed9d73 0x6ed9f80 0x6eda166 0x6eda4b5 0x6eda859 0x6edaa89 0x6edacf3 0x6edafc0 0x6edb1de 0x6edb296 0x6edb35c 0x6edb43d 0x5a39e18 0x5a39f22 0x5a39fde 0x5a3a09a 0x5a3a156 0x5a3a2a3 0x5a3a42e 0x5a3a7ef 0x5a56730 0xb882ea5 0xb89d13f 0xb93e13d 0xb93f78b 0xb61bf71 0xb61c8ed 0x367a090 0x3feaf 0x3ff5f 0x3463444 2024-04-16T07:42:14.905 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2024-04-16T07:42:19.520 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2024-04-16T07:42:19.520 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2024-04-16T07:42:19.520 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2024-04-16T07:42:19.520 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2024-04-16T07:42:19.520 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2024-04-16T07:42:19.520 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-04-16T07:42:19.520 INFO:teuthology.orchestra.run.smithi031.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid e7bebb51-b79a-49a6-ad6e-f22a47077039 2024-04-16T07:42:19.543 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T07:42:19.571 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-16T07:42:19.605 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-16T07:42:19.605 INFO:tasks.ceph:role: osd.2 2024-04-16T07:42:19.605 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com 2024-04-16T07:42:19.605 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-16T07:42:19.677 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T07:42:19.677 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T07:42:19.677 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T07:42:19.677 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T07:42:19.677 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T07:42:19.677 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-04-16T07:42:19.677 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T07:42:19.677 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T07:42:19.677 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T07:42:19.677 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T07:42:19.712 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-04-16T07:42:19.714 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-04-16T07:42:19.714 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-04-16T07:42:19.789 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-16T07:42:19.863 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-16T07:42:19.932 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-16T07:42:19.932 INFO:tasks.ceph:role: osd.3 2024-04-16T07:42:19.932 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com 2024-04-16T07:42:19.932 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-16T07:42:19.999 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T07:42:20.000 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T07:42:20.000 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T07:42:20.000 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T07:42:20.000 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T07:42:20.000 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-04-16T07:42:20.000 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T07:42:20.000 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T07:42:20.000 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T07:42:20.000 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T07:42:20.035 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-04-16T07:42:20.037 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-04-16T07:42:20.037 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-04-16T07:42:20.109 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-16T07:42:20.181 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:42:20.485 INFO:teuthology.orchestra.run.smithi094.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:42:20.486 INFO:teuthology.orchestra.run.smithi094.stderr:INFO 2024-04-16 07:42:20,488 seastar - Reactor backend: linux-aio 2024-04-16T07:42:20.498 INFO:teuthology.orchestra.run.smithi094.stderr:==28826==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:42:20.620 INFO:teuthology.orchestra.run.smithi094.stderr:INFO 2024-04-16 07:42:20,622 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T07:42:20.620 INFO:teuthology.orchestra.run.smithi094.stderr:INFO 2024-04-16 07:42:20,622 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-04-16T07:42:20.621 INFO:teuthology.orchestra.run.smithi094.stderr:DEBUG 2024-04-16 07:42:20,623 [shard 0:main] seastar - reactor::drain 2024-04-16T07:42:21.158 INFO:teuthology.orchestra.run.smithi094.stderr:----------------------------------------------------- 2024-04-16T07:42:21.159 INFO:teuthology.orchestra.run.smithi094.stderr:Suppressions used: 2024-04-16T07:42:21.159 INFO:teuthology.orchestra.run.smithi094.stderr: count bytes template 2024-04-16T07:42:21.159 INFO:teuthology.orchestra.run.smithi094.stderr: 1 8 MallocExtension::Register 2024-04-16T07:42:21.159 INFO:teuthology.orchestra.run.smithi094.stderr:----------------------------------------------------- 2024-04-16T07:42:21.159 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-04-16T07:42:21.218 INFO:teuthology.orchestra.run.smithi094.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:42:21.221 INFO:teuthology.orchestra.run.smithi094.stderr:INFO 2024-04-16 07:42:21,223 seastar - Reactor backend: linux-aio 2024-04-16T07:42:21.230 INFO:teuthology.orchestra.run.smithi094.stderr:==28824==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:42:21.295 INFO:teuthology.orchestra.run.smithi094.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb99d759 0xb870220 0xb87179c 0xb8719b2 0xb871b08 0xb871fd1 0x54daf 0x368cc74 0x43cefa8 0x5dafedd 0x5db010e 0x5d8c8a5 0x5bd1519 0x5bec5dd 0x6ec6e84 0x6ed8657 0x6ed9af4 0x6ed9b79 0x6ed9c7b 0x6ed9d73 0x6ed9f80 0x6eda166 0x6eda4b5 0x6eda859 0x6edaa89 0x6edacf3 0x6edafc0 0x6edb1de 0x6edb296 0x6edb35c 0x6edb43d 0x5a39e18 0x5a39f22 0x5a39fde 0x5a3a09a 0x5a3a156 0x5a3a2a3 0x5a3a42e 0x5a3a7ef 0x5a56730 0xb882ea5 0xb89d13f 0xb93e13d 0xb93f78b 0xb61bf71 0xb61c8ed 0x367a090 0x3feaf 0x3ff5f 0x3463444 2024-04-16T07:42:21.295 INFO:teuthology.orchestra.run.smithi094.stderr:kernel callstack: 2024-04-16T07:42:25.857 INFO:teuthology.orchestra.run.smithi094.stderr:----------------------------------------------------- 2024-04-16T07:42:25.857 INFO:teuthology.orchestra.run.smithi094.stderr:Suppressions used: 2024-04-16T07:42:25.858 INFO:teuthology.orchestra.run.smithi094.stderr: count bytes template 2024-04-16T07:42:25.858 INFO:teuthology.orchestra.run.smithi094.stderr: 1 8 MallocExtension::Register 2024-04-16T07:42:25.858 INFO:teuthology.orchestra.run.smithi094.stderr:----------------------------------------------------- 2024-04-16T07:42:25.858 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-04-16T07:42:25.858 INFO:teuthology.orchestra.run.smithi094.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid e7bebb51-b79a-49a6-ad6e-f22a47077039 2024-04-16T07:42:25.882 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T07:42:25.910 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:42:26.210 INFO:teuthology.orchestra.run.smithi094.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:42:26.211 INFO:teuthology.orchestra.run.smithi094.stderr:INFO 2024-04-16 07:42:26,213 seastar - Reactor backend: linux-aio 2024-04-16T07:42:26.216 INFO:teuthology.orchestra.run.smithi094.stderr:==29008==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:42:26.339 INFO:teuthology.orchestra.run.smithi094.stderr:INFO 2024-04-16 07:42:26,341 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T07:42:26.340 INFO:teuthology.orchestra.run.smithi094.stderr:INFO 2024-04-16 07:42:26,341 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-04-16T07:42:26.340 INFO:teuthology.orchestra.run.smithi094.stderr:DEBUG 2024-04-16 07:42:26,342 [shard 0:main] seastar - reactor::drain 2024-04-16T07:42:26.870 INFO:teuthology.orchestra.run.smithi094.stderr:----------------------------------------------------- 2024-04-16T07:42:26.870 INFO:teuthology.orchestra.run.smithi094.stderr:Suppressions used: 2024-04-16T07:42:26.870 INFO:teuthology.orchestra.run.smithi094.stderr: count bytes template 2024-04-16T07:42:26.870 INFO:teuthology.orchestra.run.smithi094.stderr: 1 8 MallocExtension::Register 2024-04-16T07:42:26.870 INFO:teuthology.orchestra.run.smithi094.stderr:----------------------------------------------------- 2024-04-16T07:42:26.871 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-04-16T07:42:26.934 INFO:teuthology.orchestra.run.smithi094.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:42:26.937 INFO:teuthology.orchestra.run.smithi094.stderr:INFO 2024-04-16 07:42:26,939 seastar - Reactor backend: linux-aio 2024-04-16T07:42:26.952 INFO:teuthology.orchestra.run.smithi094.stderr:==29006==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:42:31.597 INFO:teuthology.orchestra.run.smithi094.stderr:----------------------------------------------------- 2024-04-16T07:42:31.598 INFO:teuthology.orchestra.run.smithi094.stderr:Suppressions used: 2024-04-16T07:42:31.598 INFO:teuthology.orchestra.run.smithi094.stderr: count bytes template 2024-04-16T07:42:31.598 INFO:teuthology.orchestra.run.smithi094.stderr: 1 8 MallocExtension::Register 2024-04-16T07:42:31.598 INFO:teuthology.orchestra.run.smithi094.stderr:----------------------------------------------------- 2024-04-16T07:42:31.598 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-04-16T07:42:31.598 INFO:teuthology.orchestra.run.smithi094.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid e7bebb51-b79a-49a6-ad6e-f22a47077039 2024-04-16T07:42:31.623 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T07:42:31.651 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-16T07:42:31.652 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-16T07:42:31.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-16T07:42:31.685 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-16T07:42:31.685 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-16T07:42:31.758 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-16T07:42:31.758 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-16T07:42:31.786 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-16T07:42:31.786 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-16T07:42:31.854 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-16T07:42:31.854 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-16T07:42:31.923 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-16T07:42:31.923 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-16T07:42:31.942 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-16T07:42:31.942 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-16T07:42:31.978 INFO:tasks.ceph:Adding keys to all mons... 2024-04-16T07:42:31.978 DEBUG:teuthology.orchestra.run.smithi031:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-16T07:42:32.013 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.0] 2024-04-16T07:42:32.013 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDRKx5mOjDUCxAAHzrI+phOK20GOToC8yrI4g== 2024-04-16T07:42:32.013 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.1] 2024-04-16T07:42:32.013 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDXKx5mjHcFABAA7FsVz79MNDPwexwPsSQ+TA== 2024-04-16T07:42:32.013 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.x] 2024-04-16T07:42:32.014 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDOKx5mpP2rEBAAKjBq9Z/98P0N8+Kiw23o/A== 2024-04-16T07:42:32.014 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.2] 2024-04-16T07:42:32.014 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDdKx5mwjqNFhAAzO4aj/oaDCNxK7r9dKO3ww== 2024-04-16T07:42:32.014 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.3] 2024-04-16T07:42:32.014 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDjKx5mQlT+BRAAfZCjitXULvPJuGT9ChbKhg== 2024-04-16T07:42:32.014 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-04-16T07:42:32.015 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDOKx5mg4oIFBAAGc82YpQy94ORDV8xh0hdEw== 2024-04-16T07:42:32.015 INFO:teuthology.orchestra.run.smithi031.stdout:[client.1] 2024-04-16T07:42:32.015 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDOKx5m6CkrGBAATzZxXV4CTxvxygAAAny/DQ== 2024-04-16T07:42:32.016 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T07:42:32.126 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T07:42:32.191 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-16T07:42:32.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T07:42:32.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T07:42:32.381 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T07:42:32.447 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T07:42:32.513 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-16T07:42:32.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-16T07:42:32.541 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-16T07:42:32.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:32.666+0000 7fb2202a8a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:32.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:32.666+0000 7fb2202a8a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:32.679 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-16T07:42:32.707 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:42:32.762 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-16T07:42:32.762 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-16T07:42:32.762 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-16T07:42:32.804 INFO:tasks.ceph.mon.a:Started 2024-04-16T07:42:32.804 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-16T07:42:32.805 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-16T07:42:32.805 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-16T07:42:32.807 INFO:tasks.ceph.mgr.x:Started 2024-04-16T07:42:32.807 INFO:tasks.ceph:Setting crush tunables to default 2024-04-16T07:42:32.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd crush tunables default 2024-04-16T07:42:32.890 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-04-16T07:42:32.890+0000 7f0630437a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:32.890 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-04-16T07:42:32.890+0000 7f0630437a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:32.890 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-04-16T07:42:32.890+0000 7f0630437a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:32.913 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:32.913+0000 7ff8b7a2c000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:32.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:32.917+0000 7fa18975d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:32.924 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:32.924+0000 7ff8b7a2c000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:32.924 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:32.924+0000 7ff8b7a2c000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:32.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:32.928+0000 7fa18975d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:33.002 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2024-04-16T07:42:33.013 INFO:tasks.ceph:check_enable_crimson: True 2024-04-16T07:42:33.013 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-16T07:42:33.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-16T07:42:33.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:33.121+0000 7f5903ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:33.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:33.130+0000 7f5903ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:33.213 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-04-16T07:42:33.214+0000 7f0626df4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-16T07:42:33.213 INFO:tasks.ceph.mon.a.smithi031.stderr:Please be aware that this feature is experimental, untested, 2024-04-16T07:42:33.213 INFO:tasks.ceph.mon.a.smithi031.stderr:unsupported, and may result in data corruption, data loss, 2024-04-16T07:42:33.213 INFO:tasks.ceph.mon.a.smithi031.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-16T07:42:33.214 INFO:tasks.ceph.mon.a.smithi031.stderr:feature with important data. 2024-04-16T07:42:33.214 INFO:tasks.ceph.mon.a.smithi031.stderr: 2024-04-16T07:42:33.391 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:33.390+0000 7ff8b7a2c000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T07:42:33.444 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:33.444+0000 7ff8b7a2c000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T07:42:33.976 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:33.976+0000 7ff8b7a2c000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T07:42:34.018 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-16T07:42:34.018 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-16T07:42:34.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-16T07:42:34.030 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:34.029+0000 7ff8b7a2c000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T07:42:34.050 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-16T07:42:34.050 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-16T07:42:34.074 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:34.074+0000 7ff8b7a2c000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T07:42:34.121 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-16T07:42:34.121 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-16T07:42:34.128 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:34.127+0000 7ff8b7a2c000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T07:42:34.150 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-16T07:42:34.150 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-16T07:42:34.173 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:34.172+0000 7ff8b7a2c000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T07:42:34.200 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 084a14f9-5250-4e4c-b301-d83d55d78456 0 2024-04-16T07:42:34.295 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:34.295+0000 7ff8b7a2c000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T07:42:34.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-16T07:42:34.346+0000 7fe5ea59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:34.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-16T07:42:34.355+0000 7fe5ea59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:34.371 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:34.370+0000 7ff8b7a2c000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T07:42:34.418 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:34.418+0000 7ff8b7a2c000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T07:42:34.435 INFO:teuthology.orchestra.run.smithi094.stdout:0 2024-04-16T07:42:34.447 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 57dfea50-dece-44ba-a179-94d1d8031287 1 2024-04-16T07:42:34.474 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:34.473+0000 7ff8b7a2c000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T07:42:34.523 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:34.523+0000 7ff8b7a2c000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T07:42:34.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-16T07:42:34.552+0000 7f8a9426d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:34.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-16T07:42:34.561+0000 7f8a9426d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:34.641 INFO:teuthology.orchestra.run.smithi094.stdout:1 2024-04-16T07:42:34.652 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 7c61c7ab-b852-4236-b7d6-67efb22b14a8 2 2024-04-16T07:42:34.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-16T07:42:34.799+0000 7f0aadeed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:34.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-16T07:42:34.808+0000 7f0aadeed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:34.888 INFO:teuthology.orchestra.run.smithi094.stdout:2 2024-04-16T07:42:34.900 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 4c0e00d0-7aef-460c-93ac-36e9f9b69051 3 2024-04-16T07:42:34.920 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:34.920+0000 7ff8b7a2c000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T07:42:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-16T07:42:35.006+0000 7fa06b624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:35.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-16T07:42:35.015+0000 7fa06b624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:35.067 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:35.067+0000 7ff8b7a2c000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T07:42:35.110 INFO:teuthology.orchestra.run.smithi094.stdout:3 2024-04-16T07:42:35.121 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-16T07:42:35.160 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-16T07:42:35.160 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-16T07:42:35.187 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:35.187+0000 7ff8b7a2c000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T07:42:35.203 INFO:tasks.ceph.osd.0:Started 2024-04-16T07:42:35.203 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-16T07:42:35.233 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:35.232+0000 7ff8b7a2c000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T07:42:35.238 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-16T07:42:35.239 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-16T07:42:35.280 INFO:tasks.ceph.osd.1:Started 2024-04-16T07:42:35.281 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-16T07:42:35.314 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-16T07:42:35.315 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-16T07:42:35.349 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:35.349+0000 7ff8b7a2c000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T07:42:35.351 INFO:tasks.ceph.osd.2:Started 2024-04-16T07:42:35.351 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-04-16T07:42:35.384 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-16T07:42:35.384 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-16T07:42:35.426 INFO:tasks.ceph.osd.3:Started 2024-04-16T07:42:35.427 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:42:35.456 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:35.455+0000 7ff8b7a2c000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T07:42:35.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:35.545+0000 7f9386464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:35.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:35.553+0000 7f9386464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:35.624 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:35.624 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":7,"fsid":"e7bebb51-b79a-49a6-ad6e-f22a47077039","created":"2024-04-16T07:42:32.913621+0000","modified":"2024-04-16T07:42:35.110685+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T07:42:35.110685+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"084a14f9-5250-4e4c-b301-d83d55d78456","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"57dfea50-dece-44ba-a179-94d1d8031287","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7c61c7ab-b852-4236-b7d6-67efb22b14a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4c0e00d0-7aef-460c-93ac-36e9f9b69051","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:42:35.630 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:35.630+0000 7ff8b7a2c000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T07:42:35.635 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-16T07:42:35.635 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-16T07:42:35.677 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:35.677+0000 7ff8b7a2c000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T07:42:35.842 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:35.842+0000 7ff8b7a2c000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T07:42:36.038 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T07:42:36.145 INFO:teuthology.misc.health.smithi031.stderr:2024-04-16T07:42:36.141+0000 7f8818a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:36.149 INFO:teuthology.misc.health.smithi031.stderr:2024-04-16T07:42:36.150+0000 7f8818a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:36.220 INFO:teuthology.misc.health.smithi031.stdout: 2024-04-16T07:42:36.220 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":7,"fsid":"e7bebb51-b79a-49a6-ad6e-f22a47077039","created":"2024-04-16T07:42:32.913621+0000","modified":"2024-04-16T07:42:35.110685+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T07:42:35.110685+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"084a14f9-5250-4e4c-b301-d83d55d78456","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"57dfea50-dece-44ba-a179-94d1d8031287","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7c61c7ab-b852-4236-b7d6-67efb22b14a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4c0e00d0-7aef-460c-93ac-36e9f9b69051","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:42:36.231 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-04-16T07:42:36.735 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:36.735+0000 7ff8b7a2c000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T07:42:37.022 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:37.023+0000 7ff8b7a2c000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T07:42:37.081 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:37.081+0000 7ff8b7a2c000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T07:42:37.209 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:37.209+0000 7ff8b7a2c000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T07:42:38.218 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:42:38.218+0000 7ff8567fc640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-16T07:42:42.635 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T07:42:42.752 INFO:teuthology.misc.health.smithi031.stderr:2024-04-16T07:42:42.749+0000 7fb636c24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:42.757 INFO:teuthology.misc.health.smithi031.stderr:2024-04-16T07:42:42.758+0000 7fb636c24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:42.958 INFO:teuthology.misc.health.smithi031.stdout: 2024-04-16T07:42:42.958 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":10,"fsid":"e7bebb51-b79a-49a6-ad6e-f22a47077039","created":"2024-04-16T07:42:32.913621+0000","modified":"2024-04-16T07:42:42.229959+0000","last_up_change":"2024-04-16T07:42:40.219679+0000","last_in_change":"2024-04-16T07:42:35.110685+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:42:41.235890+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"084a14f9-5250-4e4c-b301-d83d55d78456","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6801","nonce":3418295052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3418295052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3418295052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":3418295052}]},"public_addr":"172.21.15.31:6801/3418295052","cluster_addr":"172.21.15.31:6802/3418295052","heartbeat_back_addr":"172.21.15.31:6808/3418295052","heartbeat_front_addr":"172.21.15.31:6807/3418295052","state":["exists","up"]},{"osd":1,"uuid":"57dfea50-dece-44ba-a179-94d1d8031287","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2610221363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":2610221363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2610221363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":2610221363}]},"public_addr":"172.21.15.31:6804/2610221363","cluster_addr":"172.21.15.31:6805/2610221363","heartbeat_back_addr":"172.21.15.31:6806/2610221363","heartbeat_front_addr":"172.21.15.31:6803/2610221363","state":["exists","up"]},{"osd":2,"uuid":"7c61c7ab-b852-4236-b7d6-67efb22b14a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6805","nonce":361559937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":361559937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":361559937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":361559937}]},"public_addr":"172.21.15.94:6805/361559937","cluster_addr":"172.21.15.94:6804/361559937","heartbeat_back_addr":"172.21.15.94:6809/361559937","heartbeat_front_addr":"172.21.15.94:6808/361559937","state":["exists","up"]},{"osd":3,"uuid":"4c0e00d0-7aef-460c-93ac-36e9f9b69051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3685649189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6807","nonce":3685649189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6803","nonce":3685649189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3685649189}]},"public_addr":"172.21.15.94:6806/3685649189","cluster_addr":"172.21.15.94:6807/3685649189","heartbeat_back_addr":"172.21.15.94:6803/3685649189","heartbeat_front_addr":"172.21.15.94:6802/3685649189","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:42:42.969 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-04-16T07:42:42.969 INFO:tasks.ceph:Creating RBD pool 2024-04-16T07:42:42.969 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-16T07:42:43.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:43.079+0000 7f30bb2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:43.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:43.091+0000 7f30bb2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:44.239 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'rbd' created 2024-04-16T07:42:44.257 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-16T07:42:44.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:44.386+0000 7fc45bb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:44.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:44.401+0000 7fc45bb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:45.243 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-16T07:42:45.266 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-16T07:42:45.266 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-16T07:42:45.266 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-16T07:42:45.266 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-16T07:42:45.267 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-16T07:42:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:45.425+0000 7f186b554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:45.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:45.434+0000 7f186b554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:45.692 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:45.703 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":662084667},{"type":"v1","addr":"172.21.15.94:6801","nonce":662084667}]},"active_addr":"172.21.15.94:6801/662084667","active_change":"2024-04-16T07:42:37.212106+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":1122905565}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":3914517991}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":362159873}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":3414151834}]}]} 2024-04-16T07:42:45.708 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-16T07:42:45.708 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-16T07:42:45.708 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:42:45.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:45.821+0000 7f7d32f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:45.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:45.829+0000 7f7d32f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.028 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:46.028 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"fsid":"e7bebb51-b79a-49a6-ad6e-f22a47077039","created":"2024-04-16T07:42:32.913621+0000","modified":"2024-04-16T07:42:45.241484+0000","last_up_change":"2024-04-16T07:42:40.219679+0000","last_in_change":"2024-04-16T07:42:35.110685+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:42:41.235890+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T07:42:43.297387+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"084a14f9-5250-4e4c-b301-d83d55d78456","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6801","nonce":3418295052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3418295052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3418295052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":3418295052}]},"public_addr":"172.21.15.31:6801/3418295052","cluster_addr":"172.21.15.31:6802/3418295052","heartbeat_back_addr":"172.21.15.31:6808/3418295052","heartbeat_front_addr":"172.21.15.31:6807/3418295052","state":["exists","up"]},{"osd":1,"uuid":"57dfea50-dece-44ba-a179-94d1d8031287","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2610221363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":2610221363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2610221363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":2610221363}]},"public_addr":"172.21.15.31:6804/2610221363","cluster_addr":"172.21.15.31:6805/2610221363","heartbeat_back_addr":"172.21.15.31:6806/2610221363","heartbeat_front_addr":"172.21.15.31:6803/2610221363","state":["exists","up"]},{"osd":2,"uuid":"7c61c7ab-b852-4236-b7d6-67efb22b14a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6805","nonce":361559937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":361559937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":361559937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":361559937}]},"public_addr":"172.21.15.94:6805/361559937","cluster_addr":"172.21.15.94:6804/361559937","heartbeat_back_addr":"172.21.15.94:6809/361559937","heartbeat_front_addr":"172.21.15.94:6808/361559937","state":["exists","up"]},{"osd":3,"uuid":"4c0e00d0-7aef-460c-93ac-36e9f9b69051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3685649189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6807","nonce":3685649189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6803","nonce":3685649189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3685649189}]},"public_addr":"172.21.15.94:6806/3685649189","cluster_addr":"172.21.15.94:6807/3685649189","heartbeat_back_addr":"172.21.15.94:6803/3685649189","heartbeat_front_addr":"172.21.15.94:6802/3685649189","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:42:46.041 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-16T07:42:46.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:42:46.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:46.153+0000 7fe0b226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:46.161+0000 7fe0b226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.363 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:46.363 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":14,"fsid":"e7bebb51-b79a-49a6-ad6e-f22a47077039","created":"2024-04-16T07:42:32.913621+0000","modified":"2024-04-16T07:42:46.254308+0000","last_up_change":"2024-04-16T07:42:40.219679+0000","last_in_change":"2024-04-16T07:42:35.110685+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:42:41.235890+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T07:42:43.297387+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"084a14f9-5250-4e4c-b301-d83d55d78456","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6801","nonce":3418295052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3418295052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3418295052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":3418295052}]},"public_addr":"172.21.15.31:6801/3418295052","cluster_addr":"172.21.15.31:6802/3418295052","heartbeat_back_addr":"172.21.15.31:6808/3418295052","heartbeat_front_addr":"172.21.15.31:6807/3418295052","state":["exists","up"]},{"osd":1,"uuid":"57dfea50-dece-44ba-a179-94d1d8031287","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2610221363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":2610221363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2610221363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":2610221363}]},"public_addr":"172.21.15.31:6804/2610221363","cluster_addr":"172.21.15.31:6805/2610221363","heartbeat_back_addr":"172.21.15.31:6806/2610221363","heartbeat_front_addr":"172.21.15.31:6803/2610221363","state":["exists","up"]},{"osd":2,"uuid":"7c61c7ab-b852-4236-b7d6-67efb22b14a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6805","nonce":361559937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":361559937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":361559937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":361559937}]},"public_addr":"172.21.15.94:6805/361559937","cluster_addr":"172.21.15.94:6804/361559937","heartbeat_back_addr":"172.21.15.94:6809/361559937","heartbeat_front_addr":"172.21.15.94:6808/361559937","state":["exists","up"]},{"osd":3,"uuid":"4c0e00d0-7aef-460c-93ac-36e9f9b69051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3685649189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6807","nonce":3685649189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6803","nonce":3685649189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3685649189}]},"public_addr":"172.21.15.94:6806/3685649189","cluster_addr":"172.21.15.94:6807/3685649189","heartbeat_back_addr":"172.21.15.94:6803/3685649189","heartbeat_front_addr":"172.21.15.94:6802/3685649189","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:42:46.377 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-16T07:42:46.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-16T07:42:46.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-16T07:42:46.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-16T07:42:46.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:46.501+0000 7f748b66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:46.501+0000 7f554d4ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:46.501+0000 7fecc76ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:46.502+0000 7fdd6659d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:46.512+0000 7f554d4ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:46.512+0000 7fecc76ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:46.512+0000 7f748b66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:46.513+0000 7fdd6659d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.546 INFO:teuthology.orchestra.run.smithi031.stdout:38654705670 2024-04-16T07:42:46.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-16T07:42:46.549 INFO:teuthology.orchestra.run.smithi031.stdout:38654705670 2024-04-16T07:42:46.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T07:42:46.553 INFO:teuthology.orchestra.run.smithi031.stdout:38654705670 2024-04-16T07:42:46.553 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T07:42:46.554 INFO:teuthology.orchestra.run.smithi031.stdout:38654705670 2024-04-16T07:42:46.555 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T07:42:46.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:46.665+0000 7fef95c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:46.680+0000 7fef95c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:46.713+0000 7f782022d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:46.713+0000 7fc2f706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:46.717+0000 7fc5870ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:46.723+0000 7f782022d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:46.724+0000 7fc2f706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:46.728+0000 7fc5870ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:46.887 INFO:teuthology.orchestra.run.smithi031.stdout:38654705669 2024-04-16T07:42:46.900 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.3 2024-04-16T07:42:46.926 INFO:teuthology.orchestra.run.smithi031.stdout:38654705669 2024-04-16T07:42:46.942 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.2 2024-04-16T07:42:46.953 INFO:teuthology.orchestra.run.smithi031.stdout:38654705669 2024-04-16T07:42:46.965 INFO:teuthology.orchestra.run.smithi031.stdout:38654705669 2024-04-16T07:42:46.967 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.1 2024-04-16T07:42:46.979 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.0 2024-04-16T07:42:47.902 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-16T07:42:47.943 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T07:42:47.968 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T07:42:47.979 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T07:42:48.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:48.031+0000 7fb50782d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:48.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:48.044+0000 7fb50782d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:48.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:48.075+0000 7f9b1059d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:48.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:48.087+0000 7f9b1059d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:48.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:48.115+0000 7f5a11a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:48.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:48.126+0000 7f5a11a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:48.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:48.137+0000 7f82eaf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:48.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:48.149+0000 7f82eaf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:48.296 INFO:teuthology.orchestra.run.smithi031.stdout:38654705669 2024-04-16T07:42:48.314 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.2 2024-04-16T07:42:48.327 INFO:teuthology.orchestra.run.smithi031.stdout:38654705669 2024-04-16T07:42:48.327 INFO:teuthology.orchestra.run.smithi031.stdout:38654705669 2024-04-16T07:42:48.342 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.0 2024-04-16T07:42:48.342 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.3 2024-04-16T07:42:48.386 INFO:teuthology.orchestra.run.smithi031.stdout:38654705669 2024-04-16T07:42:48.399 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.1 2024-04-16T07:42:49.315 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T07:42:49.343 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T07:42:49.343 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-16T07:42:49.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T07:42:49.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:49.449+0000 7fe01cf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:49.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:49.462+0000 7fe01cf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:49.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:49.486+0000 7fd89755d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:49.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:49.488+0000 7f9cb322d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:49.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:49.498+0000 7fd89755d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:49.500+0000 7f9cb322d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:49.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:49.531+0000 7fa88f06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:49.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:49.541+0000 7fa88f06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:49.680 INFO:teuthology.orchestra.run.smithi031.stdout:38654705669 2024-04-16T07:42:49.693 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.2 2024-04-16T07:42:49.741 INFO:teuthology.orchestra.run.smithi031.stdout:38654705669 2024-04-16T07:42:49.755 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.1 2024-04-16T07:42:49.757 INFO:teuthology.orchestra.run.smithi031.stdout:38654705669 2024-04-16T07:42:49.768 INFO:teuthology.orchestra.run.smithi031.stdout:38654705669 2024-04-16T07:42:49.770 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.0 2024-04-16T07:42:49.781 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.3 2024-04-16T07:42:50.694 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T07:42:50.756 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T07:42:50.771 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T07:42:50.781 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-16T07:42:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:50.821+0000 7f4f56ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:50.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:50.833+0000 7f4f56ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:50.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:50.897+0000 7efd6a217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:50.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:50.907+0000 7efd6a217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:50.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:50.921+0000 7f05f6206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:50.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:50.922+0000 7f7db2d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:50.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:50.931+0000 7f05f6206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:50.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:50.934+0000 7f7db2d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:51.083 INFO:teuthology.orchestra.run.smithi031.stdout:38654705669 2024-04-16T07:42:51.098 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.2 2024-04-16T07:42:51.134 INFO:teuthology.orchestra.run.smithi031.stdout:38654705669 2024-04-16T07:42:51.135 INFO:teuthology.orchestra.run.smithi031.stdout:38654705669 2024-04-16T07:42:51.142 INFO:teuthology.orchestra.run.smithi031.stdout:38654705669 2024-04-16T07:42:51.149 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.1 2024-04-16T07:42:51.154 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.3 2024-04-16T07:42:51.159 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.0 2024-04-16T07:42:52.099 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T07:42:52.150 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T07:42:52.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-16T07:42:52.160 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T07:42:52.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:52.227+0000 7f790026d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:52.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:52.239+0000 7f790026d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:52.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:52.274+0000 7f7f2c6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:52.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:52.286+0000 7f7f2c6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:52.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:52.317+0000 7f5df526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:52.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:52.327+0000 7f5df526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:52.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:52.348+0000 7f8ddfeed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:52.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:52.359+0000 7f8ddfeed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:52.496 INFO:teuthology.orchestra.run.smithi031.stdout:38654705674 2024-04-16T07:42:52.506 INFO:teuthology.orchestra.run.smithi031.stdout:38654705674 2024-04-16T07:42:52.509 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705674 for osd.1 2024-04-16T07:42:52.509 DEBUG:teuthology.parallel:result is None 2024-04-16T07:42:52.519 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705674 for osd.2 2024-04-16T07:42:52.519 DEBUG:teuthology.parallel:result is None 2024-04-16T07:42:52.530 INFO:teuthology.orchestra.run.smithi031.stdout:38654705674 2024-04-16T07:42:52.543 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705674 for osd.3 2024-04-16T07:42:52.543 DEBUG:teuthology.parallel:result is None 2024-04-16T07:42:52.582 INFO:teuthology.orchestra.run.smithi031.stdout:38654705674 2024-04-16T07:42:52.594 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705674 for osd.0 2024-04-16T07:42:52.594 DEBUG:teuthology.parallel:result is None 2024-04-16T07:42:52.595 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T07:42:52.595 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:42:52.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:52.746+0000 7f104831d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:52.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:52.754+0000 7f104831d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:52.947 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:52.947 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-16T07:42:52.959 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-16T07:42:51.217910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107952,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379408,"statfs":{"total":386547056640,"available":386436513792,"internally_reserved":0,"allocated":634880,"data_stored":1883310,"data_compressed":89856,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.964919"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.374760+0000","last_change":"2024-04-16T07:42:45.376121+0000","last_active":"2024-04-16T07:42:45.374760+0000","last_peered":"2024-04-16T07:42:45.374760+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.370390+0000","last_became_peered":"2024-04-16T07:42:45.370390+0000","last_unstale":"2024-04-16T07:42:45.374760+0000","last_undegraded":"2024-04-16T07:42:45.374760+0000","last_fullsized":"2024-04-16T07:42:45.374760+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.331165+0000","last_change":"2024-04-16T07:42:45.333004+0000","last_active":"2024-04-16T07:42:45.331165+0000","last_peered":"2024-04-16T07:42:45.331165+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.323737+0000","last_became_peered":"2024-04-16T07:42:45.323737+0000","last_unstale":"2024-04-16T07:42:45.331165+0000","last_undegraded":"2024-04-16T07:42:45.331165+0000","last_fullsized":"2024-04-16T07:42:45.331165+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:42:46.275422+0000","last_change":"2024-04-16T07:42:45.332152+0000","last_active":"2024-04-16T07:42:46.275422+0000","last_peered":"2024-04-16T07:42:46.275422+0000","last_clean":"2024-04-16T07:42:46.275422+0000","last_became_active":"2024-04-16T07:42:45.325833+0000","last_became_peered":"2024-04-16T07:42:45.325833+0000","last_unstale":"2024-04-16T07:42:46.275422+0000","last_undegraded":"2024-04-16T07:42:46.275422+0000","last_fullsized":"2024-04-16T07:42:46.275422+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.339473+0000","last_change":"2024-04-16T07:42:45.340605+0000","last_active":"2024-04-16T07:42:45.339473+0000","last_peered":"2024-04-16T07:42:45.339473+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.335238+0000","last_became_peered":"2024-04-16T07:42:45.335238+0000","last_unstale":"2024-04-16T07:42:45.339473+0000","last_undegraded":"2024-04-16T07:42:45.339473+0000","last_fullsized":"2024-04-16T07:42:45.339473+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.356397+0000","last_change":"2024-04-16T07:42:45.357843+0000","last_active":"2024-04-16T07:42:45.356397+0000","last_peered":"2024-04-16T07:42:45.356397+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.348908+0000","last_became_peered":"2024-04-16T07:42:45.348908+0000","last_unstale":"2024-04-16T07:42:45.356397+0000","last_undegraded":"2024-04-16T07:42:45.356397+0000","last_fullsized":"2024-04-16T07:42:45.356397+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.325226+0000","last_change":"2024-04-16T07:42:45.327681+0000","last_active":"2024-04-16T07:42:45.325226+0000","last_peered":"2024-04-16T07:42:45.325226+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.318385+0000","last_became_peered":"2024-04-16T07:42:45.318385+0000","last_unstale":"2024-04-16T07:42:45.325226+0000","last_undegraded":"2024-04-16T07:42:45.325226+0000","last_fullsized":"2024-04-16T07:42:45.325226+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.333328+0000","last_change":"2024-04-16T07:42:45.334636+0000","last_active":"2024-04-16T07:42:45.333328+0000","last_peered":"2024-04-16T07:42:45.333328+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.329643+0000","last_became_peered":"2024-04-16T07:42:45.329643+0000","last_unstale":"2024-04-16T07:42:45.333328+0000","last_undegraded":"2024-04-16T07:42:45.333328+0000","last_fullsized":"2024-04-16T07:42:45.333328+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.348344+0000","last_change":"2024-04-16T07:42:45.350220+0000","last_active":"2024-04-16T07:42:45.348344+0000","last_peered":"2024-04-16T07:42:45.348344+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.343569+0000","last_became_peered":"2024-04-16T07:42:45.343569+0000","last_unstale":"2024-04-16T07:42:45.348344+0000","last_undegraded":"2024-04-16T07:42:45.348344+0000","last_fullsized":"2024-04-16T07:42:45.348344+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:42:46.271689+0000","last_change":"2024-04-16T07:42:43.287780+0000","last_active":"2024-04-16T07:42:46.271689+0000","last_peered":"2024-04-16T07:42:46.271689+0000","last_clean":"2024-04-16T07:42:46.271689+0000","last_became_active":"2024-04-16T07:42:43.282375+0000","last_became_peered":"2024-04-16T07:42:43.282375+0000","last_unstale":"2024-04-16T07:42:46.271689+0000","last_undegraded":"2024-04-16T07:42:46.271689+0000","last_fullsized":"2024-04-16T07:42:46.271689+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_clean_scrub_stamp":"2024-04-16T07:42:42.229959+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:42:52.960 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:42:53.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:53.071+0000 7f1b17ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:53.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:53.080+0000 7f1b17ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:53.275 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:53.276 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-16T07:42:53.288 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-16T07:42:53.218316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107952,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379408,"statfs":{"total":386547056640,"available":386436513792,"internally_reserved":0,"allocated":634880,"data_stored":1883310,"data_compressed":89856,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.965325"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.374760+0000","last_change":"2024-04-16T07:42:45.376121+0000","last_active":"2024-04-16T07:42:45.374760+0000","last_peered":"2024-04-16T07:42:45.374760+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.370390+0000","last_became_peered":"2024-04-16T07:42:45.370390+0000","last_unstale":"2024-04-16T07:42:45.374760+0000","last_undegraded":"2024-04-16T07:42:45.374760+0000","last_fullsized":"2024-04-16T07:42:45.374760+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.331165+0000","last_change":"2024-04-16T07:42:45.333004+0000","last_active":"2024-04-16T07:42:45.331165+0000","last_peered":"2024-04-16T07:42:45.331165+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.323737+0000","last_became_peered":"2024-04-16T07:42:45.323737+0000","last_unstale":"2024-04-16T07:42:45.331165+0000","last_undegraded":"2024-04-16T07:42:45.331165+0000","last_fullsized":"2024-04-16T07:42:45.331165+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:42:46.275422+0000","last_change":"2024-04-16T07:42:45.332152+0000","last_active":"2024-04-16T07:42:46.275422+0000","last_peered":"2024-04-16T07:42:46.275422+0000","last_clean":"2024-04-16T07:42:46.275422+0000","last_became_active":"2024-04-16T07:42:45.325833+0000","last_became_peered":"2024-04-16T07:42:45.325833+0000","last_unstale":"2024-04-16T07:42:46.275422+0000","last_undegraded":"2024-04-16T07:42:46.275422+0000","last_fullsized":"2024-04-16T07:42:46.275422+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.339473+0000","last_change":"2024-04-16T07:42:45.340605+0000","last_active":"2024-04-16T07:42:45.339473+0000","last_peered":"2024-04-16T07:42:45.339473+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.335238+0000","last_became_peered":"2024-04-16T07:42:45.335238+0000","last_unstale":"2024-04-16T07:42:45.339473+0000","last_undegraded":"2024-04-16T07:42:45.339473+0000","last_fullsized":"2024-04-16T07:42:45.339473+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.356397+0000","last_change":"2024-04-16T07:42:45.357843+0000","last_active":"2024-04-16T07:42:45.356397+0000","last_peered":"2024-04-16T07:42:45.356397+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.348908+0000","last_became_peered":"2024-04-16T07:42:45.348908+0000","last_unstale":"2024-04-16T07:42:45.356397+0000","last_undegraded":"2024-04-16T07:42:45.356397+0000","last_fullsized":"2024-04-16T07:42:45.356397+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.325226+0000","last_change":"2024-04-16T07:42:45.327681+0000","last_active":"2024-04-16T07:42:45.325226+0000","last_peered":"2024-04-16T07:42:45.325226+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.318385+0000","last_became_peered":"2024-04-16T07:42:45.318385+0000","last_unstale":"2024-04-16T07:42:45.325226+0000","last_undegraded":"2024-04-16T07:42:45.325226+0000","last_fullsized":"2024-04-16T07:42:45.325226+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.333328+0000","last_change":"2024-04-16T07:42:45.334636+0000","last_active":"2024-04-16T07:42:45.333328+0000","last_peered":"2024-04-16T07:42:45.333328+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.329643+0000","last_became_peered":"2024-04-16T07:42:45.329643+0000","last_unstale":"2024-04-16T07:42:45.333328+0000","last_undegraded":"2024-04-16T07:42:45.333328+0000","last_fullsized":"2024-04-16T07:42:45.333328+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.348344+0000","last_change":"2024-04-16T07:42:45.350220+0000","last_active":"2024-04-16T07:42:45.348344+0000","last_peered":"2024-04-16T07:42:45.348344+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.343569+0000","last_became_peered":"2024-04-16T07:42:45.343569+0000","last_unstale":"2024-04-16T07:42:45.348344+0000","last_undegraded":"2024-04-16T07:42:45.348344+0000","last_fullsized":"2024-04-16T07:42:45.348344+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:42:46.271689+0000","last_change":"2024-04-16T07:42:43.287780+0000","last_active":"2024-04-16T07:42:46.271689+0000","last_peered":"2024-04-16T07:42:46.271689+0000","last_clean":"2024-04-16T07:42:46.271689+0000","last_became_active":"2024-04-16T07:42:43.282375+0000","last_became_peered":"2024-04-16T07:42:43.282375+0000","last_unstale":"2024-04-16T07:42:46.271689+0000","last_undegraded":"2024-04-16T07:42:46.271689+0000","last_fullsized":"2024-04-16T07:42:46.271689+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_clean_scrub_stamp":"2024-04-16T07:42:42.229959+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:42:53.288 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-16T07:42:53.288 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-16T07:42:53.288 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-16T07:42:53.289 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-16T07:42:53.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:53.410+0000 7f45f9aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:53.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:53.417+0000 7f45f9aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:53.652 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:53.652 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-16T07:42:53.665 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-16T07:42:53.666 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-16T07:42:53.677 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-16T07:42:53.677 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-matanb-crimson-replicated-april-15'}, 'ceph': {'conf': {'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-matanb-crimson-replicated-april-15', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}} 2024-04-16T07:42:53.677 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-16T07:42:53.677 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-16T07:42:53.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:53.788+0000 7f4fba06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:53.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:53.800+0000 7f4fba06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:54.015 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-16T07:42:54.015 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T07:42:54.015 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:42:54.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:54.126+0000 7f5b522ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:54.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:54.134+0000 7f5b522ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:54.330 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:54.330 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-16T07:42:54.342 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-16T07:42:53.218316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107952,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379408,"statfs":{"total":386547056640,"available":386436513792,"internally_reserved":0,"allocated":634880,"data_stored":1883310,"data_compressed":89856,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.965325"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.374760+0000","last_change":"2024-04-16T07:42:45.376121+0000","last_active":"2024-04-16T07:42:45.374760+0000","last_peered":"2024-04-16T07:42:45.374760+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.370390+0000","last_became_peered":"2024-04-16T07:42:45.370390+0000","last_unstale":"2024-04-16T07:42:45.374760+0000","last_undegraded":"2024-04-16T07:42:45.374760+0000","last_fullsized":"2024-04-16T07:42:45.374760+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.331165+0000","last_change":"2024-04-16T07:42:45.333004+0000","last_active":"2024-04-16T07:42:45.331165+0000","last_peered":"2024-04-16T07:42:45.331165+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.323737+0000","last_became_peered":"2024-04-16T07:42:45.323737+0000","last_unstale":"2024-04-16T07:42:45.331165+0000","last_undegraded":"2024-04-16T07:42:45.331165+0000","last_fullsized":"2024-04-16T07:42:45.331165+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:42:46.275422+0000","last_change":"2024-04-16T07:42:45.332152+0000","last_active":"2024-04-16T07:42:46.275422+0000","last_peered":"2024-04-16T07:42:46.275422+0000","last_clean":"2024-04-16T07:42:46.275422+0000","last_became_active":"2024-04-16T07:42:45.325833+0000","last_became_peered":"2024-04-16T07:42:45.325833+0000","last_unstale":"2024-04-16T07:42:46.275422+0000","last_undegraded":"2024-04-16T07:42:46.275422+0000","last_fullsized":"2024-04-16T07:42:46.275422+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.339473+0000","last_change":"2024-04-16T07:42:45.340605+0000","last_active":"2024-04-16T07:42:45.339473+0000","last_peered":"2024-04-16T07:42:45.339473+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.335238+0000","last_became_peered":"2024-04-16T07:42:45.335238+0000","last_unstale":"2024-04-16T07:42:45.339473+0000","last_undegraded":"2024-04-16T07:42:45.339473+0000","last_fullsized":"2024-04-16T07:42:45.339473+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.356397+0000","last_change":"2024-04-16T07:42:45.357843+0000","last_active":"2024-04-16T07:42:45.356397+0000","last_peered":"2024-04-16T07:42:45.356397+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.348908+0000","last_became_peered":"2024-04-16T07:42:45.348908+0000","last_unstale":"2024-04-16T07:42:45.356397+0000","last_undegraded":"2024-04-16T07:42:45.356397+0000","last_fullsized":"2024-04-16T07:42:45.356397+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.325226+0000","last_change":"2024-04-16T07:42:45.327681+0000","last_active":"2024-04-16T07:42:45.325226+0000","last_peered":"2024-04-16T07:42:45.325226+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.318385+0000","last_became_peered":"2024-04-16T07:42:45.318385+0000","last_unstale":"2024-04-16T07:42:45.325226+0000","last_undegraded":"2024-04-16T07:42:45.325226+0000","last_fullsized":"2024-04-16T07:42:45.325226+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.333328+0000","last_change":"2024-04-16T07:42:45.334636+0000","last_active":"2024-04-16T07:42:45.333328+0000","last_peered":"2024-04-16T07:42:45.333328+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.329643+0000","last_became_peered":"2024-04-16T07:42:45.329643+0000","last_unstale":"2024-04-16T07:42:45.333328+0000","last_undegraded":"2024-04-16T07:42:45.333328+0000","last_fullsized":"2024-04-16T07:42:45.333328+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.348344+0000","last_change":"2024-04-16T07:42:45.350220+0000","last_active":"2024-04-16T07:42:45.348344+0000","last_peered":"2024-04-16T07:42:45.348344+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.343569+0000","last_became_peered":"2024-04-16T07:42:45.343569+0000","last_unstale":"2024-04-16T07:42:45.348344+0000","last_undegraded":"2024-04-16T07:42:45.348344+0000","last_fullsized":"2024-04-16T07:42:45.348344+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:42:46.271689+0000","last_change":"2024-04-16T07:42:43.287780+0000","last_active":"2024-04-16T07:42:46.271689+0000","last_peered":"2024-04-16T07:42:46.271689+0000","last_clean":"2024-04-16T07:42:46.271689+0000","last_became_active":"2024-04-16T07:42:43.282375+0000","last_became_peered":"2024-04-16T07:42:43.282375+0000","last_unstale":"2024-04-16T07:42:46.271689+0000","last_undegraded":"2024-04-16T07:42:46.271689+0000","last_fullsized":"2024-04-16T07:42:46.271689+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_clean_scrub_stamp":"2024-04-16T07:42:42.229959+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:42:54.343 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:42:54.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:54.455+0000 7f335466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:54.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:54.464+0000 7f335466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:54.662 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:54.663 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-16T07:42:54.675 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-16T07:42:53.218316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107952,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379408,"statfs":{"total":386547056640,"available":386436513792,"internally_reserved":0,"allocated":634880,"data_stored":1883310,"data_compressed":89856,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.965325"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.374760+0000","last_change":"2024-04-16T07:42:45.376121+0000","last_active":"2024-04-16T07:42:45.374760+0000","last_peered":"2024-04-16T07:42:45.374760+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.370390+0000","last_became_peered":"2024-04-16T07:42:45.370390+0000","last_unstale":"2024-04-16T07:42:45.374760+0000","last_undegraded":"2024-04-16T07:42:45.374760+0000","last_fullsized":"2024-04-16T07:42:45.374760+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.331165+0000","last_change":"2024-04-16T07:42:45.333004+0000","last_active":"2024-04-16T07:42:45.331165+0000","last_peered":"2024-04-16T07:42:45.331165+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.323737+0000","last_became_peered":"2024-04-16T07:42:45.323737+0000","last_unstale":"2024-04-16T07:42:45.331165+0000","last_undegraded":"2024-04-16T07:42:45.331165+0000","last_fullsized":"2024-04-16T07:42:45.331165+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:42:46.275422+0000","last_change":"2024-04-16T07:42:45.332152+0000","last_active":"2024-04-16T07:42:46.275422+0000","last_peered":"2024-04-16T07:42:46.275422+0000","last_clean":"2024-04-16T07:42:46.275422+0000","last_became_active":"2024-04-16T07:42:45.325833+0000","last_became_peered":"2024-04-16T07:42:45.325833+0000","last_unstale":"2024-04-16T07:42:46.275422+0000","last_undegraded":"2024-04-16T07:42:46.275422+0000","last_fullsized":"2024-04-16T07:42:46.275422+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.339473+0000","last_change":"2024-04-16T07:42:45.340605+0000","last_active":"2024-04-16T07:42:45.339473+0000","last_peered":"2024-04-16T07:42:45.339473+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.335238+0000","last_became_peered":"2024-04-16T07:42:45.335238+0000","last_unstale":"2024-04-16T07:42:45.339473+0000","last_undegraded":"2024-04-16T07:42:45.339473+0000","last_fullsized":"2024-04-16T07:42:45.339473+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.356397+0000","last_change":"2024-04-16T07:42:45.357843+0000","last_active":"2024-04-16T07:42:45.356397+0000","last_peered":"2024-04-16T07:42:45.356397+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.348908+0000","last_became_peered":"2024-04-16T07:42:45.348908+0000","last_unstale":"2024-04-16T07:42:45.356397+0000","last_undegraded":"2024-04-16T07:42:45.356397+0000","last_fullsized":"2024-04-16T07:42:45.356397+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.325226+0000","last_change":"2024-04-16T07:42:45.327681+0000","last_active":"2024-04-16T07:42:45.325226+0000","last_peered":"2024-04-16T07:42:45.325226+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.318385+0000","last_became_peered":"2024-04-16T07:42:45.318385+0000","last_unstale":"2024-04-16T07:42:45.325226+0000","last_undegraded":"2024-04-16T07:42:45.325226+0000","last_fullsized":"2024-04-16T07:42:45.325226+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.333328+0000","last_change":"2024-04-16T07:42:45.334636+0000","last_active":"2024-04-16T07:42:45.333328+0000","last_peered":"2024-04-16T07:42:45.333328+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.329643+0000","last_became_peered":"2024-04-16T07:42:45.329643+0000","last_unstale":"2024-04-16T07:42:45.333328+0000","last_undegraded":"2024-04-16T07:42:45.333328+0000","last_fullsized":"2024-04-16T07:42:45.333328+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.348344+0000","last_change":"2024-04-16T07:42:45.350220+0000","last_active":"2024-04-16T07:42:45.348344+0000","last_peered":"2024-04-16T07:42:45.348344+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.343569+0000","last_became_peered":"2024-04-16T07:42:45.343569+0000","last_unstale":"2024-04-16T07:42:45.348344+0000","last_undegraded":"2024-04-16T07:42:45.348344+0000","last_fullsized":"2024-04-16T07:42:45.348344+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:42:46.271689+0000","last_change":"2024-04-16T07:42:43.287780+0000","last_active":"2024-04-16T07:42:46.271689+0000","last_peered":"2024-04-16T07:42:46.271689+0000","last_clean":"2024-04-16T07:42:46.271689+0000","last_became_active":"2024-04-16T07:42:43.282375+0000","last_became_peered":"2024-04-16T07:42:43.282375+0000","last_unstale":"2024-04-16T07:42:46.271689+0000","last_undegraded":"2024-04-16T07:42:46.271689+0000","last_fullsized":"2024-04-16T07:42:46.271689+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_clean_scrub_stamp":"2024-04-16T07:42:42.229959+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:42:54.676 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-16T07:42:54.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-16T07:42:54.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:54.788+0000 7fcfdf75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:54.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:54.797+0000 7fcfdf75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:54.996 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 14 2024-04-16T07:42:54.996 INFO:teuthology.orchestra.run.smithi031.stdout:fsid e7bebb51-b79a-49a6-ad6e-f22a47077039 2024-04-16T07:42:54.996 INFO:teuthology.orchestra.run.smithi031.stdout:created 2024-04-16T07:42:32.913621+0000 2024-04-16T07:42:54.996 INFO:teuthology.orchestra.run.smithi031.stdout:modified 2024-04-16T07:42:46.254308+0000 2024-04-16T07:42:54.996 INFO:teuthology.orchestra.run.smithi031.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-16T07:42:54.996 INFO:teuthology.orchestra.run.smithi031.stdout:crush_version 3 2024-04-16T07:42:54.996 INFO:teuthology.orchestra.run.smithi031.stdout:full_ratio 0.95 2024-04-16T07:42:54.996 INFO:teuthology.orchestra.run.smithi031.stdout:backfillfull_ratio 0.9 2024-04-16T07:42:54.996 INFO:teuthology.orchestra.run.smithi031.stdout:nearfull_ratio 0.85 2024-04-16T07:42:54.996 INFO:teuthology.orchestra.run.smithi031.stdout:require_min_compat_client luminous 2024-04-16T07:42:54.996 INFO:teuthology.orchestra.run.smithi031.stdout:min_compat_client jewel 2024-04-16T07:42:54.996 INFO:teuthology.orchestra.run.smithi031.stdout:require_osd_release squid 2024-04-16T07:42:54.997 INFO:teuthology.orchestra.run.smithi031.stdout:stretch_mode_enabled false 2024-04-16T07:42:54.997 INFO:teuthology.orchestra.run.smithi031.stdout:allow_crimson=true 2024-04-16T07:42:54.997 INFO:teuthology.orchestra.run.smithi031.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 12 flags hashpspool,nopgchange,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2024-04-16T07:42:54.997 INFO:teuthology.orchestra.run.smithi031.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 14 flags hashpspool,nopgchange,crimson stripe_width 0 application rbd read_balance_score 1.50 2024-04-16T07:42:54.997 INFO:teuthology.orchestra.run.smithi031.stdout:max_osd 4 2024-04-16T07:42:54.997 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.31:6801/3418295052 v2:172.21.15.31:6802/3418295052 exists,up 084a14f9-5250-4e4c-b301-d83d55d78456 2024-04-16T07:42:54.997 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.31:6804/2610221363 v2:172.21.15.31:6805/2610221363 exists,up 57dfea50-dece-44ba-a179-94d1d8031287 2024-04-16T07:42:54.997 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.94:6805/361559937 v2:172.21.15.94:6804/361559937 exists,up 7c61c7ab-b852-4236-b7d6-67efb22b14a8 2024-04-16T07:42:54.997 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.94:6806/3685649189 v2:172.21.15.94:6807/3685649189 exists,up 4c0e00d0-7aef-460c-93ac-36e9f9b69051 2024-04-16T07:42:55.009 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.31:6801/3418295052 v2:172.21.15.31:6802/3418295052 exists,up 084a14f9-5250-4e4c-b301-d83d55d78456', 'osd.1 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.31:6804/2610221363 v2:172.21.15.31:6805/2610221363 exists,up 57dfea50-dece-44ba-a179-94d1d8031287', 'osd.2 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.94:6805/361559937 v2:172.21.15.94:6804/361559937 exists,up 7c61c7ab-b852-4236-b7d6-67efb22b14a8', 'osd.3 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.94:6806/3685649189 v2:172.21.15.94:6807/3685649189 exists,up 4c0e00d0-7aef-460c-93ac-36e9f9b69051'] 2024-04-16T07:42:55.009 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-16T07:42:55.120 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-16T07:42:55.120 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mon.a", 2024-04-16T07:42:55.120 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster": "ceph", 2024-04-16T07:42:55.120 INFO:teuthology.orchestra.run.smithi031.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-16T07:42:55.120 INFO:teuthology.orchestra.run.smithi031.stdout: "admin_socket_mode": "", 2024-04-16T07:42:55.120 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-16T07:42:55.120 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_client_required": "cephx, none", 2024-04-16T07:42:55.120 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_cluster_required": "cephx", 2024-04-16T07:42:55.121 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_debug": "true", 2024-04-16T07:42:55.121 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-16T07:42:55.121 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-16T07:42:55.121 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_service_required": "cephx", 2024-04-16T07:42:55.121 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-16T07:42:55.121 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_supported": "cephx", 2024-04-16T07:42:55.121 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_aio": "true", 2024-04-16T07:42:55.121 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-16T07:42:55.121 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_aio_poll_ms": "250", 2024-04-16T07:42:55.121 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_aio_reap_max": "16", 2024-04-16T07:42:55.121 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_async_discard_threads": "0", 2024-04-16T07:42:55.121 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_block_size": "4096", 2024-04-16T07:42:55.121 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_debug_aio": "false", 2024-04-16T07:42:55.121 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-16T07:42:55.122 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-16T07:42:55.122 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_debug_inflight_ios": "false", 2024-04-16T07:42:55.122 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_enable_discard": "false", 2024-04-16T07:42:55.122 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_flock_retry": "3", 2024-04-16T07:42:55.122 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-16T07:42:55.122 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_inject_crash": "0", 2024-04-16T07:42:55.122 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-16T07:42:55.122 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_ioring": "false", 2024-04-16T07:42:55.122 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_ioring_hipri": "false", 2024-04-16T07:42:55.122 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-16T07:42:55.122 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-16T07:42:55.122 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-16T07:42:55.122 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-16T07:42:55.122 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_type": "", 2024-04-16T07:42:55.123 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_alloc_size": "1048576", 2024-04-16T07:42:55.123 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_allocator": "hybrid", 2024-04-16T07:42:55.123 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_buffered_io": "true", 2024-04-16T07:42:55.123 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_check_for_zeros": "false", 2024-04-16T07:42:55.123 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-16T07:42:55.123 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-16T07:42:55.123 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_compact_log_sync": "false", 2024-04-16T07:42:55.123 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-16T07:42:55.123 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-16T07:42:55.123 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-16T07:42:55.123 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-16T07:42:55.123 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_max_log_runway": "4194304", 2024-04-16T07:42:55.123 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_max_prefetch": "1048576", 2024-04-16T07:42:55.123 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_min_flush_size": "524288", 2024-04-16T07:42:55.124 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_min_log_runway": "1048576", 2024-04-16T07:42:55.124 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_replay_recovery": "false", 2024-04-16T07:42:55.124 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-16T07:42:55.124 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-16T07:42:55.124 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_sync_write": "false", 2024-04-16T07:42:55.124 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-16T07:42:55.124 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-16T07:42:55.124 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-16T07:42:55.124 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_allocation_from_file": "true", 2024-04-16T07:42:55.124 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_allocator": "hybrid", 2024-04-16T07:42:55.124 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-16T07:42:55.124 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-16T07:42:55.124 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-16T07:42:55.124 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-16T07:42:55.124 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-16T07:42:55.125 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-16T07:42:55.125 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-16T07:42:55.125 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_create": "true", 2024-04-16T07:42:55.125 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_db_create": "false", 2024-04-16T07:42:55.125 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_db_path": "", 2024-04-16T07:42:55.125 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_db_size": "0", 2024-04-16T07:42:55.125 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_path": "", 2024-04-16T07:42:55.125 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_preallocate_file": "false", 2024-04-16T07:42:55.125 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_size": "107374182400", 2024-04-16T07:42:55.125 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_wal_create": "false", 2024-04-16T07:42:55.125 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_wal_path": "", 2024-04-16T07:42:55.125 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_wal_size": "100663296", 2024-04-16T07:42:55.125 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bluefs": "true", 2024-04-16T07:42:55.125 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-16T07:42:55.126 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-16T07:42:55.126 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-16T07:42:55.126 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-16T07:42:55.126 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-16T07:42:55.126 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-16T07:42:55.126 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-16T07:42:55.126 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-16T07:42:55.126 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_autotune": "true", 2024-04-16T07:42:55.126 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-16T07:42:55.126 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-16T07:42:55.126 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-16T07:42:55.126 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-16T07:42:55.126 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_size": "0", 2024-04-16T07:42:55.126 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-16T07:42:55.127 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-16T07:42:55.127 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-16T07:42:55.127 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-16T07:42:55.127 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_type": "2q", 2024-04-16T07:42:55.127 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-16T07:42:55.127 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_clone_cow": "true", 2024-04-16T07:42:55.127 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-16T07:42:55.127 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-16T07:42:55.127 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-16T07:42:55.127 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-16T07:42:55.127 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-16T07:42:55.127 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-16T07:42:55.127 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-16T07:42:55.127 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_mode": "none", 2024-04-16T07:42:55.128 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-16T07:42:55.128 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_csum_type": "crc32c", 2024-04-16T07:42:55.128 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-04-16T07:42:55.128 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-16T07:42:55.128 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_freelist": "false", 2024-04-16T07:42:55.128 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-16T07:42:55.128 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-16T07:42:55.128 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-16T07:42:55.128 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-16T07:42:55.128 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-16T07:42:55.128 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-16T07:42:55.128 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-16T07:42:55.128 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-16T07:42:55.128 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-16T07:42:55.129 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-16T07:42:55.129 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-16T07:42:55.129 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-16T07:42:55.129 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-16T07:42:55.129 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_small_allocations": "0", 2024-04-16T07:42:55.129 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-16T07:42:55.129 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_default_buffered_read": "true", 2024-04-16T07:42:55.129 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_default_buffered_write": "false", 2024-04-16T07:42:55.129 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-16T07:42:55.129 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-16T07:42:55.129 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-16T07:42:55.129 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-16T07:42:55.129 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-16T07:42:55.129 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-16T07:42:55.129 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-16T07:42:55.130 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-16T07:42:55.130 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-16T07:42:55.130 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fail_eio": "false", 2024-04-16T07:42:55.130 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-16T07:42:55.130 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-16T07:42:55.130 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-16T07:42:55.130 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-16T07:42:55.130 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-16T07:42:55.130 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-16T07:42:55.130 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_on_mount": "false", 2024-04-16T07:42:55.130 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-16T07:42:55.130 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_on_umount": "false", 2024-04-16T07:42:55.130 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-16T07:42:55.131 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-16T07:42:55.131 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-16T07:42:55.131 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-16T07:42:55.131 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-16T07:42:55.131 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-16T07:42:55.131 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-16T07:42:55.131 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-16T07:42:55.131 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_ignore_data_csum": "false", 2024-04-16T07:42:55.131 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-16T07:42:55.131 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-16T07:42:55.131 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-16T07:42:55.131 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-16T07:42:55.131 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_log_op_age": "5.000000", 2024-04-16T07:42:55.131 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_max_alloc_size": "0", 2024-04-16T07:42:55.132 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_max_blob_size": "0", 2024-04-16T07:42:55.132 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-16T07:42:55.132 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-16T07:42:55.132 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-16T07:42:55.132 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_max_deferred_txc": "32", 2024-04-16T07:42:55.132 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_min_alloc_size": "0", 2024-04-16T07:42:55.132 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-16T07:42:55.132 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-16T07:42:55.132 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_nid_prealloc": "1024", 2024-04-16T07:42:55.132 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-16T07:42:55.132 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-16T07:42:55.132 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-16T07:42:55.132 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-16T07:42:55.132 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_retry_disk_reads": "3", 2024-04-16T07:42:55.133 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_rocksdb_cf": "true", 2024-04-16T07:42:55.133 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-16T07:42:55.133 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-16T07:42:55.133 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-16T07:42:55.133 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-16T07:42:55.133 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-16T07:42:55.133 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-16T07:42:55.133 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_spdk_mem": "512", 2024-04-16T07:42:55.133 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-16T07:42:55.133 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-16T07:42:55.133 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-16T07:42:55.133 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-16T07:42:55.133 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-16T07:42:55.133 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-16T07:42:55.134 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-16T07:42:55.134 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_tracing": "false", 2024-04-16T07:42:55.134 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-16T07:42:55.134 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-16T07:42:55.134 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-16T07:42:55.134 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-16T07:42:55.134 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-16T07:42:55.134 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-16T07:42:55.134 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-16T07:42:55.134 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-16T07:42:55.134 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-16T07:42:55.134 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_zero_block_detection": "false", 2024-04-16T07:42:55.134 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-16T07:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-16T07:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-16T07:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-16T07:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-16T07:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-16T07:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-16T07:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-16T07:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-16T07:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-16T07:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-16T07:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-16T07:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-16T07:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-16T07:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_cluster_require_signatures": "false", 2024-04-16T07:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_cluster_require_version": "2", 2024-04-16T07:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_require_signatures": "false", 2024-04-16T07:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_require_version": "2", 2024-04-16T07:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_service_require_signatures": "false", 2024-04-16T07:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_service_require_version": "2", 2024-04-16T07:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_sign_messages": "true", 2024-04-16T07:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "chdir": "", 2024-04-16T07:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "client_acl_type": "", 2024-04-16T07:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "client_asio_thread_count": "2", 2024-04-16T07:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "client_cache_mid": "0.750000", 2024-04-16T07:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "client_cache_size": "16384", 2024-04-16T07:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "client_caps_release_delay": "5", 2024-04-16T07:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "client_check_pool_perm": "true", 2024-04-16T07:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-16T07:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "client_debug_force_sync_read": "false", 2024-04-16T07:42:55.137 INFO:teuthology.orchestra.run.smithi031.stdout: "client_debug_getattr_caps": "false", 2024-04-16T07:42:55.137 INFO:teuthology.orchestra.run.smithi031.stdout: "client_debug_inject_tick_delay": "0", 2024-04-16T07:42:55.137 INFO:teuthology.orchestra.run.smithi031.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-16T07:42:55.137 INFO:teuthology.orchestra.run.smithi031.stdout: "client_die_on_failed_remount": "false", 2024-04-16T07:42:55.137 INFO:teuthology.orchestra.run.smithi031.stdout: "client_dirsize_rbytes": "true", 2024-04-16T07:42:55.137 INFO:teuthology.orchestra.run.smithi031.stdout: "client_force_lazyio": "false", 2024-04-16T07:42:55.137 INFO:teuthology.orchestra.run.smithi031.stdout: "client_fs": "", 2024-04-16T07:42:55.137 INFO:teuthology.orchestra.run.smithi031.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-16T07:42:55.137 INFO:teuthology.orchestra.run.smithi031.stdout: "client_inject_release_failure": "false", 2024-04-16T07:42:55.137 INFO:teuthology.orchestra.run.smithi031.stdout: "client_max_inline_size": "4096", 2024-04-16T07:42:55.137 INFO:teuthology.orchestra.run.smithi031.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-16T07:42:55.137 INFO:teuthology.orchestra.run.smithi031.stdout: "client_mds_namespace": "", 2024-04-16T07:42:55.137 INFO:teuthology.orchestra.run.smithi031.stdout: "client_metadata": "", 2024-04-16T07:42:55.137 INFO:teuthology.orchestra.run.smithi031.stdout: "client_mount_gid": "-1", 2024-04-16T07:42:55.137 INFO:teuthology.orchestra.run.smithi031.stdout: "client_mount_timeout": "300", 2024-04-16T07:42:55.138 INFO:teuthology.orchestra.run.smithi031.stdout: "client_mount_uid": "-1", 2024-04-16T07:42:55.138 INFO:teuthology.orchestra.run.smithi031.stdout: "client_mountpoint": "/", 2024-04-16T07:42:55.138 INFO:teuthology.orchestra.run.smithi031.stdout: "client_notify_timeout": "10", 2024-04-16T07:42:55.138 INFO:teuthology.orchestra.run.smithi031.stdout: "client_oc": "true", 2024-04-16T07:42:55.138 INFO:teuthology.orchestra.run.smithi031.stdout: "client_oc_max_dirty": "104857600", 2024-04-16T07:42:55.138 INFO:teuthology.orchestra.run.smithi031.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-16T07:42:55.138 INFO:teuthology.orchestra.run.smithi031.stdout: "client_oc_max_objects": "1000", 2024-04-16T07:42:55.138 INFO:teuthology.orchestra.run.smithi031.stdout: "client_oc_size": "209715200", 2024-04-16T07:42:55.138 INFO:teuthology.orchestra.run.smithi031.stdout: "client_oc_target_dirty": "8388608", 2024-04-16T07:42:55.138 INFO:teuthology.orchestra.run.smithi031.stdout: "client_permissions": "true", 2024-04-16T07:42:55.138 INFO:teuthology.orchestra.run.smithi031.stdout: "client_quota": "true", 2024-04-16T07:42:55.138 INFO:teuthology.orchestra.run.smithi031.stdout: "client_quota_df": "true", 2024-04-16T07:42:55.138 INFO:teuthology.orchestra.run.smithi031.stdout: "client_readahead_max_bytes": "0", 2024-04-16T07:42:55.138 INFO:teuthology.orchestra.run.smithi031.stdout: "client_readahead_max_periods": "4", 2024-04-16T07:42:55.139 INFO:teuthology.orchestra.run.smithi031.stdout: "client_readahead_min": "131072", 2024-04-16T07:42:55.139 INFO:teuthology.orchestra.run.smithi031.stdout: "client_reconnect_stale": "false", 2024-04-16T07:42:55.139 INFO:teuthology.orchestra.run.smithi031.stdout: "client_shutdown_timeout": "30", 2024-04-16T07:42:55.139 INFO:teuthology.orchestra.run.smithi031.stdout: "client_snapdir": ".snap", 2024-04-16T07:42:55.139 INFO:teuthology.orchestra.run.smithi031.stdout: "client_tick_interval": "1", 2024-04-16T07:42:55.139 INFO:teuthology.orchestra.run.smithi031.stdout: "client_trace": "", 2024-04-16T07:42:55.139 INFO:teuthology.orchestra.run.smithi031.stdout: "client_try_dentry_invalidate": "false", 2024-04-16T07:42:55.139 INFO:teuthology.orchestra.run.smithi031.stdout: "client_use_faked_inos": "false", 2024-04-16T07:42:55.139 INFO:teuthology.orchestra.run.smithi031.stdout: "client_use_random_mds": "false", 2024-04-16T07:42:55.139 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_graylog": "false", 2024-04-16T07:42:55.139 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-16T07:42:55.139 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_graylog_port": "12201", 2024-04-16T07:42:55.139 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_monitors": "default=true", 2024-04-16T07:42:55.139 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_syslog": "false", 2024-04-16T07:42:55.139 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-16T07:42:55.140 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_syslog_level": "info", 2024-04-16T07:42:55.140 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "-", 2024-04-16T07:42:55.140 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_network": "", 2024-04-16T07:42:55.140 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_network_interface": "", 2024-04-16T07:42:55.140 INFO:teuthology.orchestra.run.smithi031.stdout: "compressor_zlib_isal": "false", 2024-04-16T07:42:55.140 INFO:teuthology.orchestra.run.smithi031.stdout: "compressor_zlib_level": "5", 2024-04-16T07:42:55.140 INFO:teuthology.orchestra.run.smithi031.stdout: "compressor_zlib_winsize": "-15", 2024-04-16T07:42:55.140 INFO:teuthology.orchestra.run.smithi031.stdout: "compressor_zstd_level": "1", 2024-04-16T07:42:55.140 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-16T07:42:55.140 INFO:teuthology.orchestra.run.smithi031.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-16T07:42:55.140 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_alien_op_num_threads": "6", 2024-04-16T07:42:55.140 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-16T07:42:55.140 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-16T07:42:55.140 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-16T07:42:55.140 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_osd_stat_interval": "0", 2024-04-16T07:42:55.141 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_seastar_cpu_cores": "", 2024-04-16T07:42:55.141 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_location": "", 2024-04-16T07:42:55.141 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_location_hook": "", 2024-04-16T07:42:55.141 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_location_hook_timeout": "10", 2024-04-16T07:42:55.141 INFO:teuthology.orchestra.run.smithi031.stdout: "daemonize": "false", 2024-04-16T07:42:55.141 INFO:teuthology.orchestra.run.smithi031.stdout: "daos_pool": "tank", 2024-04-16T07:42:55.141 INFO:teuthology.orchestra.run.smithi031.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-16T07:42:55.141 INFO:teuthology.orchestra.run.smithi031.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-16T07:42:55.141 INFO:teuthology.orchestra.run.smithi031.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-16T07:42:55.141 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_alienstore": "0/5", 2024-04-16T07:42:55.141 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_allow_any_pool_priority": "false", 2024-04-16T07:42:55.141 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_asok": "1/5", 2024-04-16T07:42:55.141 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_asok_assert_abort": "false", 2024-04-16T07:42:55.141 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_asserts_on_shutdown": "true", 2024-04-16T07:42:55.141 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_auth": "20/20", 2024-04-16T07:42:55.141 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_bdev": "1/3", 2024-04-16T07:42:55.142 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_bluefs": "1/5", 2024-04-16T07:42:55.142 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_bluestore": "1/5", 2024-04-16T07:42:55.142 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_buffer": "0/1", 2024-04-16T07:42:55.142 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_ceph_exporter": "1/5", 2024-04-16T07:42:55.142 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_cephfs_mirror": "0/5", 2024-04-16T07:42:55.142 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_cephsqlite": "0/5", 2024-04-16T07:42:55.142 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_client": "0/5", 2024-04-16T07:42:55.142 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_compressor": "1/5", 2024-04-16T07:42:55.142 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_context": "0/1", 2024-04-16T07:42:55.142 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_crimson_interrupt": "0/5", 2024-04-16T07:42:55.142 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_crush": "1/1", 2024-04-16T07:42:55.142 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_crypto": "1/5", 2024-04-16T07:42:55.142 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_cyanstore": "0/5", 2024-04-16T07:42:55.142 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_deliberately_leak_memory": "false", 2024-04-16T07:42:55.143 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_disable_randomized_ping": "false", 2024-04-16T07:42:55.143 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_dpdk": "1/5", 2024-04-16T07:42:55.143 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_eventtrace": "1/5", 2024-04-16T07:42:55.143 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_filer": "0/1", 2024-04-16T07:42:55.143 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_filestore": "1/3", 2024-04-16T07:42:55.143 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_finisher": "1/1", 2024-04-16T07:42:55.143 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_fuse": "1/5", 2024-04-16T07:42:55.143 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_heartbeat_testing_span": "0", 2024-04-16T07:42:55.143 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_heartbeatmap": "1/5", 2024-04-16T07:42:55.143 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-16T07:42:55.143 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_javaclient": "1/5", 2024-04-16T07:42:55.143 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_journal": "1/3", 2024-04-16T07:42:55.143 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_journaler": "0/5", 2024-04-16T07:42:55.143 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_kstore": "1/5", 2024-04-16T07:42:55.143 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_lockdep": "0/1", 2024-04-16T07:42:55.144 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mclock": "1/5", 2024-04-16T07:42:55.144 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds": "1/5", 2024-04-16T07:42:55.144 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds_balancer": "1/5", 2024-04-16T07:42:55.144 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds_locker": "1/5", 2024-04-16T07:42:55.144 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds_log": "1/5", 2024-04-16T07:42:55.144 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds_log_expire": "1/5", 2024-04-16T07:42:55.144 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds_migrator": "1/5", 2024-04-16T07:42:55.144 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds_quiesce": "3/5", 2024-04-16T07:42:55.144 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_memstore": "1/5", 2024-04-16T07:42:55.144 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mgr": "2/5", 2024-04-16T07:42:55.144 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mgrc": "1/5", 2024-04-16T07:42:55.144 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mon": "20/20", 2024-04-16T07:42:55.144 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_monc": "0/10", 2024-04-16T07:42:55.144 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_ms": "1/1", 2024-04-16T07:42:55.145 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_none": "0/5", 2024-04-16T07:42:55.145 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_objclass": "0/5", 2024-04-16T07:42:55.145 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_objectcacher": "0/5", 2024-04-16T07:42:55.145 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_objecter": "0/1", 2024-04-16T07:42:55.145 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_optracker": "0/5", 2024-04-16T07:42:55.145 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_osd": "1/5", 2024-04-16T07:42:55.145 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_paxos": "20/20", 2024-04-16T07:42:55.145 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_perfcounter": "1/5", 2024-04-16T07:42:55.145 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_prioritycache": "1/5", 2024-04-16T07:42:55.145 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rados": "0/5", 2024-04-16T07:42:55.145 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rbd": "0/5", 2024-04-16T07:42:55.145 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rbd_mirror": "0/5", 2024-04-16T07:42:55.145 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rbd_pwl": "0/5", 2024-04-16T07:42:55.145 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rbd_replay": "0/5", 2024-04-16T07:42:55.145 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_refs": "0/0", 2024-04-16T07:42:55.146 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_reserver": "1/1", 2024-04-16T07:42:55.146 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw": "1/5", 2024-04-16T07:42:55.146 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_access": "1/5", 2024-04-16T07:42:55.146 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_datacache": "1/5", 2024-04-16T07:42:55.146 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_dbstore": "1/5", 2024-04-16T07:42:55.146 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_flight": "1/5", 2024-04-16T07:42:55.146 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-16T07:42:55.146 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_notification": "1/5", 2024-04-16T07:42:55.146 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_sync": "1/5", 2024-04-16T07:42:55.146 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rocksdb": "4/5", 2024-04-16T07:42:55.146 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore": "0/5", 2024-04-16T07:42:55.146 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_backref": "0/5", 2024-04-16T07:42:55.147 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_cache": "0/5", 2024-04-16T07:42:55.147 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_cleaner": "0/5", 2024-04-16T07:42:55.147 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_device": "0/5", 2024-04-16T07:42:55.147 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_epm": "0/5", 2024-04-16T07:42:55.147 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-16T07:42:55.147 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_journal": "0/5", 2024-04-16T07:42:55.147 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_lba": "0/5", 2024-04-16T07:42:55.147 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_odata": "0/5", 2024-04-16T07:42:55.147 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_omap": "0/5", 2024-04-16T07:42:55.147 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_onode": "0/5", 2024-04-16T07:42:55.147 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_t": "0/5", 2024-04-16T07:42:55.147 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_tm": "0/5", 2024-04-16T07:42:55.148 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_striper": "0/1", 2024-04-16T07:42:55.148 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_test": "0/5", 2024-04-16T07:42:55.148 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_throttle": "1/1", 2024-04-16T07:42:55.148 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_timer": "0/1", 2024-04-16T07:42:55.148 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_tp": "0/5", 2024-04-16T07:42:55.148 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_trace": "1/5", 2024-04-16T07:42:55.148 INFO:teuthology.orchestra.run.smithi031.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-16T07:42:55.148 INFO:teuthology.orchestra.run.smithi031.stdout: "device_failure_prediction_mode": "none", 2024-04-16T07:42:55.148 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2024-04-16T07:42:55.148 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-16T07:42:55.148 INFO:teuthology.orchestra.run.smithi031.stdout: "err_to_graylog": "false", 2024-04-16T07:42:55.148 INFO:teuthology.orchestra.run.smithi031.stdout: "err_to_journald": "false", 2024-04-16T07:42:55.148 INFO:teuthology.orchestra.run.smithi031.stdout: "err_to_stderr": "true", 2024-04-16T07:42:55.149 INFO:teuthology.orchestra.run.smithi031.stdout: "err_to_syslog": "false", 2024-04-16T07:42:55.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event_tracing": "false", 2024-04-16T07:42:55.149 INFO:teuthology.orchestra.run.smithi031.stdout: "exporter_addr": "0.0.0.0", 2024-04-16T07:42:55.149 INFO:teuthology.orchestra.run.smithi031.stdout: "exporter_http_port": "9926", 2024-04-16T07:42:55.149 INFO:teuthology.orchestra.run.smithi031.stdout: "exporter_prio_limit": "5", 2024-04-16T07:42:55.149 INFO:teuthology.orchestra.run.smithi031.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-16T07:42:55.149 INFO:teuthology.orchestra.run.smithi031.stdout: "exporter_sort_metrics": "true", 2024-04-16T07:42:55.149 INFO:teuthology.orchestra.run.smithi031.stdout: "exporter_stats_period": "5", 2024-04-16T07:42:55.149 INFO:teuthology.orchestra.run.smithi031.stdout: "fake_statfs_for_testing": "0", 2024-04-16T07:42:55.149 INFO:teuthology.orchestra.run.smithi031.stdout: "fatal_signal_handlers": "true", 2024-04-16T07:42:55.149 INFO:teuthology.orchestra.run.smithi031.stdout: "filer_max_purge_ops": "10", 2024-04-16T07:42:55.149 INFO:teuthology.orchestra.run.smithi031.stdout: "filer_max_truncate_ops": "128", 2024-04-16T07:42:55.149 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_apply_finisher_threads": "1", 2024-04-16T07:42:55.149 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_blackhole": "false", 2024-04-16T07:42:55.150 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_btrfs_clone_range": "true", 2024-04-16T07:42:55.150 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_btrfs_snap": "true", 2024-04-16T07:42:55.150 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_caller_concurrency": "10", 2024-04-16T07:42:55.150 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_collect_device_partition_information": "true", 2024-04-16T07:42:55.150 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_commit_timeout": "600.000000", 2024-04-16T07:42:55.150 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_debug_inject_read_err": "false", 2024-04-16T07:42:55.150 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_debug_omap_check": "false", 2024-04-16T07:42:55.150 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_debug_verify_split": "false", 2024-04-16T07:42:55.150 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_dump_file": "", 2024-04-16T07:42:55.150 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-16T07:42:55.150 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-16T07:42:55.150 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fadvise": "true", 2024-04-16T07:42:55.150 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fail_eio": "true", 2024-04-16T07:42:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fd_cache_shards": "16", 2024-04-16T07:42:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fd_cache_size": "128", 2024-04-16T07:42:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fiemap": "false", 2024-04-16T07:42:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fiemap_threshold": "4096", 2024-04-16T07:42:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-16T07:42:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-16T07:42:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_inject_stall": "0", 2024-04-16T07:42:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_journal_parallel": "false", 2024-04-16T07:42:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_journal_trailing": "false", 2024-04-16T07:42:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_journal_writeahead": "false", 2024-04-16T07:42:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_kill_at": "0", 2024-04-16T07:42:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-16T07:42:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-16T07:42:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-16T07:42:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-16T07:42:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-16T07:42:55.152 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattrs": "0", 2024-04-16T07:42:55.152 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-16T07:42:55.152 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-16T07:42:55.152 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-16T07:42:55.152 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-16T07:42:55.152 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_xattr_value_size": "0", 2024-04-16T07:42:55.152 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-16T07:42:55.152 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-16T07:42:55.152 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-16T07:42:55.152 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_merge_threshold": "-10", 2024-04-16T07:42:55.152 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-16T07:42:55.152 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_odsync_write": "false", 2024-04-16T07:42:55.152 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_omap_backend": "rocksdb", 2024-04-16T07:42:55.152 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_omap_backend_path": "", 2024-04-16T07:42:55.152 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-16T07:42:55.153 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-16T07:42:55.153 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-16T07:42:55.153 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_op_thread_timeout": "60", 2024-04-16T07:42:55.153 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_op_threads": "2", 2024-04-16T07:42:55.153 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_punch_hole": "false", 2024-04-16T07:42:55.153 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-16T07:42:55.153 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-16T07:42:55.153 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-16T07:42:55.153 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-16T07:42:55.153 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-16T07:42:55.153 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-16T07:42:55.153 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-16T07:42:55.153 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-16T07:42:55.153 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-16T07:42:55.153 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_max_ops": "50", 2024-04-16T07:42:55.153 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-16T07:42:55.154 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_seek_data_hole": "false", 2024-04-16T07:42:55.154 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_sloppy_crc": "false", 2024-04-16T07:42:55.154 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-16T07:42:55.154 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_splice": "false", 2024-04-16T07:42:55.154 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_split_multiple": "2", 2024-04-16T07:42:55.154 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_split_rand_factor": "20", 2024-04-16T07:42:55.154 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_update_to": "1000", 2024-04-16T07:42:55.154 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-16T07:42:55.154 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-16T07:42:55.154 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-16T07:42:55.154 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-16T07:42:55.154 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-16T07:42:55.154 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-16T07:42:55.154 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_enable": "true", 2024-04-16T07:42:55.155 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-16T07:42:55.155 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-16T07:42:55.155 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-16T07:42:55.155 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-16T07:42:55.155 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-16T07:42:55.155 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-16T07:42:55.155 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_xfs_extsize": "false", 2024-04-16T07:42:55.155 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_zfs_snap": "false", 2024-04-16T07:42:55.155 INFO:teuthology.orchestra.run.smithi031.stdout: "fio_dir": "/tmp/fio", 2024-04-16T07:42:55.155 INFO:teuthology.orchestra.run.smithi031.stdout: "fsid": "e7bebb51-b79a-49a6-ad6e-f22a47077039", 2024-04-16T07:42:55.155 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_allow_other": "true", 2024-04-16T07:42:55.155 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_atomic_o_trunc": "true", 2024-04-16T07:42:55.155 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_big_writes": "true", 2024-04-16T07:42:55.155 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_debug": "false", 2024-04-16T07:42:55.155 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_default_permissions": "false", 2024-04-16T07:42:55.156 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_disable_pagecache": "false", 2024-04-16T07:42:55.156 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_max_write": "0", 2024-04-16T07:42:55.156 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_multithreaded": "true", 2024-04-16T07:42:55.156 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_require_active_mds": "true", 2024-04-16T07:42:55.156 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_set_user_groups": "true", 2024-04-16T07:42:55.156 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_splice_move": "true", 2024-04-16T07:42:55.156 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_splice_read": "true", 2024-04-16T07:42:55.156 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_splice_write": "true", 2024-04-16T07:42:55.156 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-16T07:42:55.156 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_use_invalidate_cb": "true", 2024-04-16T07:42:55.156 INFO:teuthology.orchestra.run.smithi031.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-16T07:42:55.156 INFO:teuthology.orchestra.run.smithi031.stdout: "gss_target_name": "ceph", 2024-04-16T07:42:55.156 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_file": "", 2024-04-16T07:42:55.156 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_inject_failure": "0", 2024-04-16T07:42:55.156 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_interval": "5", 2024-04-16T07:42:55.157 INFO:teuthology.orchestra.run.smithi031.stdout: "host": "smithi031", 2024-04-16T07:42:55.157 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-16T07:42:55.157 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-16T07:42:55.157 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-16T07:42:55.157 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-16T07:42:55.157 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-16T07:42:55.157 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-16T07:42:55.157 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-16T07:42:55.157 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-16T07:42:55.157 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-16T07:42:55.157 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-16T07:42:55.157 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-16T07:42:55.157 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-16T07:42:55.157 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-16T07:42:55.158 INFO:teuthology.orchestra.run.smithi031.stdout: "inject_early_sigterm": "false", 2024-04-16T07:42:55.158 INFO:teuthology.orchestra.run.smithi031.stdout: "jaeger_agent_port": "6799", 2024-04-16T07:42:55.158 INFO:teuthology.orchestra.run.smithi031.stdout: "jaeger_tracing_enable": "false", 2024-04-16T07:42:55.158 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_aio": "true", 2024-04-16T07:42:55.158 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_align_min_size": "65536", 2024-04-16T07:42:55.158 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_block_align": "true", 2024-04-16T07:42:55.158 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_block_size": "4096", 2024-04-16T07:42:55.158 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_dio": "true", 2024-04-16T07:42:55.158 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_discard": "false", 2024-04-16T07:42:55.158 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_force_aio": "false", 2024-04-16T07:42:55.158 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_ignore_corruption": "false", 2024-04-16T07:42:55.158 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_max_write_bytes": "10485760", 2024-04-16T07:42:55.158 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_max_write_entries": "100", 2024-04-16T07:42:55.158 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_replay_from": "0", 2024-04-16T07:42:55.158 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-16T07:42:55.159 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-16T07:42:55.159 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-16T07:42:55.159 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-16T07:42:55.159 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_write_header_frequency": "0", 2024-04-16T07:42:55.159 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_zero_on_create": "false", 2024-04-16T07:42:55.159 INFO:teuthology.orchestra.run.smithi031.stdout: "journaler_prefetch_periods": "10", 2024-04-16T07:42:55.159 INFO:teuthology.orchestra.run.smithi031.stdout: "journaler_prezero_periods": "5", 2024-04-16T07:42:55.159 INFO:teuthology.orchestra.run.smithi031.stdout: "journaler_write_head_interval": "15", 2024-04-16T07:42:55.159 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-04-16T07:42:55.159 INFO:teuthology.orchestra.run.smithi031.stdout: "keyfile": "", 2024-04-16T07:42:55.159 INFO:teuthology.orchestra.run.smithi031.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-16T07:42:55.159 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_backend": "rocksdb", 2024-04-16T07:42:55.159 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_default_stripe_size": "65536", 2024-04-16T07:42:55.159 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_fsck_on_mount": "false", 2024-04-16T07:42:55.160 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-16T07:42:55.160 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_max_bytes": "67108864", 2024-04-16T07:42:55.160 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_max_ops": "512", 2024-04-16T07:42:55.160 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_nid_prealloc": "1024", 2024-04-16T07:42:55.160 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_onode_map_size": "1024", 2024-04-16T07:42:55.160 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-16T07:42:55.160 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_sync_submit_transaction": "false", 2024-04-16T07:42:55.160 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_sync_transaction": "false", 2024-04-16T07:42:55.160 INFO:teuthology.orchestra.run.smithi031.stdout: "librados_thread_count": "2", 2024-04-16T07:42:55.160 INFO:teuthology.orchestra.run.smithi031.stdout: "lockdep": "false", 2024-04-16T07:42:55.160 INFO:teuthology.orchestra.run.smithi031.stdout: "lockdep_force_backtrace": "false", 2024-04-16T07:42:55.160 INFO:teuthology.orchestra.run.smithi031.stdout: "log_coarse_timestamps": "true", 2024-04-16T07:42:55.160 INFO:teuthology.orchestra.run.smithi031.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-16T07:42:55.161 INFO:teuthology.orchestra.run.smithi031.stdout: "log_flush_on_exit": "false", 2024-04-16T07:42:55.161 INFO:teuthology.orchestra.run.smithi031.stdout: "log_graylog_host": "127.0.0.1", 2024-04-16T07:42:55.161 INFO:teuthology.orchestra.run.smithi031.stdout: "log_graylog_port": "12201", 2024-04-16T07:42:55.161 INFO:teuthology.orchestra.run.smithi031.stdout: "log_max_new": "1000", 2024-04-16T07:42:55.161 INFO:teuthology.orchestra.run.smithi031.stdout: "log_max_recent": "10000", 2024-04-16T07:42:55.161 INFO:teuthology.orchestra.run.smithi031.stdout: "log_stderr_prefix": "", 2024-04-16T07:42:55.161 INFO:teuthology.orchestra.run.smithi031.stdout: "log_stop_at_utilization": "0.970000", 2024-04-16T07:42:55.161 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": "true", 2024-04-16T07:42:55.161 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_graylog": "false", 2024-04-16T07:42:55.161 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_journald": "false", 2024-04-16T07:42:55.161 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_stderr": "false", 2024-04-16T07:42:55.161 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_syslog": "false", 2024-04-16T07:42:55.161 INFO:teuthology.orchestra.run.smithi031.stdout: "max_rotating_auth_attempts": "10", 2024-04-16T07:42:55.162 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-16T07:42:55.162 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_action_on_write_error": "1", 2024-04-16T07:42:55.162 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_alternate_name_max": "8192", 2024-04-16T07:42:55.162 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_asio_thread_count": "2", 2024-04-16T07:42:55.162 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_export_pin": "true", 2024-04-16T07:42:55.162 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_fragment_dirs": "true", 2024-04-16T07:42:55.162 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-16T07:42:55.162 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_fragment_interval": "5", 2024-04-16T07:42:55.162 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-16T07:42:55.162 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-16T07:42:55.162 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_interval": "10", 2024-04-16T07:42:55.162 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_max": "-1", 2024-04-16T07:42:55.162 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_max_until": "-1", 2024-04-16T07:42:55.163 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_merge_size": "50", 2024-04-16T07:42:55.163 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_midchunk": "0.300000", 2024-04-16T07:42:55.163 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-16T07:42:55.163 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_min_start": "0.200000", 2024-04-16T07:42:55.163 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_minchunk": "0.001000", 2024-04-16T07:42:55.163 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_mode": "0", 2024-04-16T07:42:55.163 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_need_max": "1.200000", 2024-04-16T07:42:55.163 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_need_min": "0.800000", 2024-04-16T07:42:55.163 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_overload_epochs": "2", 2024-04-16T07:42:55.163 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-16T07:42:55.163 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-16T07:42:55.163 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_split_bits": "3", 2024-04-16T07:42:55.163 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-16T07:42:55.163 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_split_size": "10000", 2024-04-16T07:42:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-16T07:42:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_target_decay": "10.000000", 2024-04-16T07:42:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-16T07:42:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_beacon_grace": "15.000000", 2024-04-16T07:42:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_beacon_interval": "4.000000", 2024-04-16T07:42:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-16T07:42:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-16T07:42:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_mid": "0.700000", 2024-04-16T07:42:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-16T07:42:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_quiesce_delay": "0", 2024-04-16T07:42:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-16T07:42:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-16T07:42:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-16T07:42:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_release_free_interval": "10", 2024-04-16T07:42:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_reservation": "0.050000", 2024-04-16T07:42:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-16T07:42:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_trim_interval": "1", 2024-04-16T07:42:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_trim_threshold": "262144", 2024-04-16T07:42:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-16T07:42:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-16T07:42:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-16T07:42:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_client_prealloc_inos": "1000", 2024-04-16T07:42:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-16T07:42:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_connect_bootstrapping": "false", 2024-04-16T07:42:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_damage_table_max_entries": "10000", 2024-04-16T07:42:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-16T07:42:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_debug_auth_pins": "false", 2024-04-16T07:42:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_debug_frag": "false", 2024-04-16T07:42:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_debug_scatterstat": "false", 2024-04-16T07:42:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_debug_subtrees": "false", 2024-04-16T07:42:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_decay_halflife": "5.000000", 2024-04-16T07:42:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_default_dir_hash": "2", 2024-04-16T07:42:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_defer_session_stale": "true", 2024-04-16T07:42:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_deny_all_reconnect": "false", 2024-04-16T07:42:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dir_keys_per_op": "16384", 2024-04-16T07:42:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dir_max_commit_size": "10", 2024-04-16T07:42:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dir_max_entries": "0", 2024-04-16T07:42:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dir_prefetch": "true", 2024-04-16T07:42:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-16T07:42:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-16T07:42:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dump_cache_on_map": "false", 2024-04-16T07:42:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-16T07:42:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-16T07:42:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_early_reply": "true", 2024-04-16T07:42:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_enable_op_tracker": "true", 2024-04-16T07:42:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_enforce_unique_name": "true", 2024-04-16T07:42:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-16T07:42:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-16T07:42:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_export_ephemeral_random": "true", 2024-04-16T07:42:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-16T07:42:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-16T07:42:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-16T07:42:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-16T07:42:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-16T07:42:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-16T07:42:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-16T07:42:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-16T07:42:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_health_summarize_threshold": "10", 2024-04-16T07:42:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-16T07:42:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-16T07:42:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_inject_health_dummy": "false", 2024-04-16T07:42:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-16T07:42:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_inject_migrator_session_race": "false", 2024-04-16T07:42:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-16T07:42:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-16T07:42:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-16T07:42:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_join_fs": "", 2024-04-16T07:42:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_journal_format": "1", 2024-04-16T07:42:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_create_at": "0", 2024-04-16T07:42:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_export_at": "0", 2024-04-16T07:42:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_import_at": "0", 2024-04-16T07:42:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_journal_at": "0", 2024-04-16T07:42:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_journal_expire_at": "0", 2024-04-16T07:42:55.169 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_journal_replay_at": "0", 2024-04-16T07:42:55.169 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_link_at": "0", 2024-04-16T07:42:55.169 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_mdstable_at": "0", 2024-04-16T07:42:55.169 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_openc_at": "0", 2024-04-16T07:42:55.169 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_rename_at": "0", 2024-04-16T07:42:55.169 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_shutdown_at": "0", 2024-04-16T07:42:55.169 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-16T07:42:55.169 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_event_large_threshold": "524288", 2024-04-16T07:42:55.169 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_events_per_segment": "1024", 2024-04-16T07:42:55.169 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-16T07:42:55.169 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_max_events": "-1", 2024-04-16T07:42:55.169 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_max_segments": "128", 2024-04-16T07:42:55.169 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_pause": "false", 2024-04-16T07:42:55.169 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_segment_size": "0", 2024-04-16T07:42:55.169 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-16T07:42:55.170 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-16T07:42:55.170 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-16T07:42:55.170 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_trim_threshold": "128", 2024-04-16T07:42:55.170 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-16T07:42:55.170 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_warn_factor": "2.000000", 2024-04-16T07:42:55.170 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_caps_per_client": "1048576", 2024-04-16T07:42:55.170 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_completed_flushes": "100000", 2024-04-16T07:42:55.170 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_completed_requests": "100000", 2024-04-16T07:42:55.170 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_export_size": "20971520", 2024-04-16T07:42:55.170 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_file_recover": "32", 2024-04-16T07:42:55.170 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_purge_files": "64", 2024-04-16T07:42:55.170 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_purge_ops": "8192", 2024-04-16T07:42:55.170 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-16T07:42:55.171 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-16T07:42:55.171 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_snaps_per_dir": "100", 2024-04-16T07:42:55.171 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_metrics_update_interval": "2", 2024-04-16T07:42:55.171 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_min_caps_per_client": "100", 2024-04-16T07:42:55.171 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_min_caps_working_set": "10000", 2024-04-16T07:42:55.171 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-16T07:42:55.171 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_numa_node": "-1", 2024-04-16T07:42:55.171 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-16T07:42:55.171 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_op_complaint_time": "30.000000", 2024-04-16T07:42:55.171 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_op_history_duration": "600", 2024-04-16T07:42:55.171 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_op_history_size": "20", 2024-04-16T07:42:55.171 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_op_history_slow_op_size": "20", 2024-04-16T07:42:55.171 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-16T07:42:55.171 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_op_log_threshold": "5", 2024-04-16T07:42:55.172 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_ping_grace": "15", 2024-04-16T07:42:55.172 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_ping_interval": "5", 2024-04-16T07:42:55.172 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-16T07:42:55.172 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-16T07:42:55.172 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_recall_max_caps": "30000", 2024-04-16T07:42:55.172 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-16T07:42:55.172 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-16T07:42:55.172 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-16T07:42:55.172 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_recall_warning_threshold": "262144", 2024-04-16T07:42:55.172 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-16T07:42:55.172 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_replay_interval": "1.000000", 2024-04-16T07:42:55.172 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-16T07:42:55.172 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-16T07:42:55.172 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_root_ino_gid": "0", 2024-04-16T07:42:55.172 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_root_ino_uid": "0", 2024-04-16T07:42:55.173 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-16T07:42:55.173 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-16T07:42:55.173 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-16T07:42:55.173 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-16T07:42:55.173 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-16T07:42:55.173 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-16T07:42:55.173 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-16T07:42:55.173 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-16T07:42:55.173 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-16T07:42:55.173 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-16T07:42:55.173 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_shutdown_check": "0", 2024-04-16T07:42:55.173 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_skip_ino": "0", 2024-04-16T07:42:55.173 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-16T07:42:55.173 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_snap_max_uid": "4294967294", 2024-04-16T07:42:55.173 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_snap_min_uid": "0", 2024-04-16T07:42:55.174 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_snap_rstat": "false", 2024-04-16T07:42:55.174 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_standby_replay_damaged": "false", 2024-04-16T07:42:55.174 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_symlink_recovery": "true", 2024-04-16T07:42:55.174 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-16T07:42:55.174 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_thrash_exports": "0", 2024-04-16T07:42:55.174 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_thrash_fragments": "0", 2024-04-16T07:42:55.174 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_tick_interval": "5.000000", 2024-04-16T07:42:55.174 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_valgrind_exit": "false", 2024-04-16T07:42:55.174 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_verify_backtrace": "1", 2024-04-16T07:42:55.174 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_verify_scatter": "false", 2024-04-16T07:42:55.174 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-16T07:42:55.174 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_wipe_sessions": "false", 2024-04-16T07:42:55.174 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool_debug": "false", 2024-04-16T07:42:55.174 INFO:teuthology.orchestra.run.smithi031.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-16T07:42:55.175 INFO:teuthology.orchestra.run.smithi031.stdout: "memstore_device_bytes": "1073741824", 2024-04-16T07:42:55.175 INFO:teuthology.orchestra.run.smithi031.stdout: "memstore_page_set": "false", 2024-04-16T07:42:55.175 INFO:teuthology.orchestra.run.smithi031.stdout: "memstore_page_size": "65536", 2024-04-16T07:42:55.175 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_client_bytes": "134217728", 2024-04-16T07:42:55.175 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_client_messages": "512", 2024-04-16T07:42:55.175 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-16T07:42:55.175 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-16T07:42:55.175 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-16T07:42:55.175 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-16T07:42:55.175 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_disabled_modules": "", 2024-04-16T07:42:55.175 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-16T07:42:55.175 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_max_pg_creating": "1024", 2024-04-16T07:42:55.175 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_max_pg_num_change": "128", 2024-04-16T07:42:55.176 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_mds_bytes": "134217728", 2024-04-16T07:42:55.176 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_mds_messages": "128", 2024-04-16T07:42:55.176 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-16T07:42:55.176 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_mon_bytes": "134217728", 2024-04-16T07:42:55.176 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_mon_messages": "128", 2024-04-16T07:42:55.176 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_osd_bytes": "536870912", 2024-04-16T07:42:55.176 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_osd_messages": "8192", 2024-04-16T07:42:55.176 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_pool": "true", 2024-04-16T07:42:55.176 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-16T07:42:55.176 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_standby_modules": "true", 2024-04-16T07:42:55.176 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_stats_period": "5", 2024-04-16T07:42:55.176 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_stats_threshold": "5", 2024-04-16T07:42:55.176 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_tick_period": "2", 2024-04-16T07:42:55.176 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-16T07:42:55.176 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-16T07:42:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_allow_pool_delete": "true", 2024-04-16T07:42:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_allow_pool_size_one": "true", 2024-04-16T07:42:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_auth_validate_all_caps": "true", 2024-04-16T07:42:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-16T07:42:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-16T07:42:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_bytes": "104857600", 2024-04-16T07:42:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_directed_command_retry": "2", 2024-04-16T07:42:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-16T07:42:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-16T07:42:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-16T07:42:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-16T07:42:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_hunt_parallel": "3", 2024-04-16T07:42:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_log_interval": "1.000000", 2024-04-16T07:42:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-16T07:42:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_ping_interval": "10.000000", 2024-04-16T07:42:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-16T07:42:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_target_rank": "-1", 2024-04-16T07:42:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-16T07:42:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-16T07:42:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-16T07:42:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_file_level": "debug", 2024-04-16T07:42:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_file": "true", 2024-04-16T07:42:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-16T07:42:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-16T07:42:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-16T07:42:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_journald": "false", 2024-04-16T07:42:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-16T07:42:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-16T07:42:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-16T07:42:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-04-16T07:42:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_compact_on_bootstrap": "false", 2024-04-16T07:42:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_compact_on_start": "false", 2024-04-16T07:42:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_compact_on_trim": "true", 2024-04-16T07:42:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-16T07:42:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-16T07:42:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-16T07:42:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cpu_threads": "4", 2024-04-16T07:42:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_crush_min_required_version": "hammer", 2024-04-16T07:42:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_daemon_bytes": "419430400", 2024-04-16T07:42:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-16T07:42:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_data_avail_crit": "5", 2024-04-16T07:42:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_data_avail_warn": "5", 2024-04-16T07:42:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_data_size_warn": "16106127360", 2024-04-16T07:42:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-16T07:42:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-16T07:42:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_dump_json": "false", 2024-04-16T07:42:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-16T07:42:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_dump_transactions": "false", 2024-04-16T07:42:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_extra_checks": "false", 2024-04-16T07:42:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-16T07:42:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-16T07:42:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_no_require_reef": "false", 2024-04-16T07:42:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_no_require_squid": "false", 2024-04-16T07:42:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-16T07:42:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-16T07:42:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-16T07:42:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_down_mkfs_grace": "120", 2024-04-16T07:42:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_down_uptime_grace": "60", 2024-04-16T07:42:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_election_default_strategy": "1", 2024-04-16T07:42:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_election_timeout": "5.000000", 2024-04-16T07:42:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-16T07:42:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_elector_ping_divisor": "2", 2024-04-16T07:42:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-16T07:42:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_enable_op_tracker": "true", 2024-04-16T07:42:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_fake_pool_delete": "false", 2024-04-16T07:42:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_force_quorum_join": "false", 2024-04-16T07:42:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_globalid_prealloc": "10000", 2024-04-16T07:42:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_health_detail_to_clog": "false", 2024-04-16T07:42:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_health_log_update_period": "5", 2024-04-16T07:42:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_health_max_detail": "50", 2024-04-16T07:42:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_health_to_clog": "true", 2024-04-16T07:42:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_health_to_clog_interval": "600", 2024-04-16T07:42:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-16T07:42:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_host": "172.21.15.31", 2024-04-16T07:42:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_host_override": "", 2024-04-16T07:42:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_initial_members": "", 2024-04-16T07:42:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-16T07:42:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-16T07:42:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-16T07:42:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-16T07:42:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-16T07:42:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_lease": "5.000000", 2024-04-16T07:42:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_log_full_interval": "50", 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_log_max": "10000", 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_log_max_summary": "50", 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_log_epochs": "500", 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_osd": "10000", 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_pg_per_osd": "10000", 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_pool_pg_num": "65536", 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mds_force_trim_to": "0", 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mds_skip_sanity": "false", 2024-04-16T07:42:55.184 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_memory_autotune": "true", 2024-04-16T07:42:55.184 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_memory_target": "2147483648", 2024-04-16T07:42:55.184 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mgr_beacon_grace": "30", 2024-04-16T07:42:55.184 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-16T07:42:55.184 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mgr_digest_period": "5", 2024-04-16T07:42:55.184 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mgr_inactive_grace": "60", 2024-04-16T07:42:55.184 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-16T07:42:55.184 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-16T07:42:55.184 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_min_osdmap_epochs": "50", 2024-04-16T07:42:55.184 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_op_complaint_time": "30", 2024-04-16T07:42:55.184 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_op_history_duration": "600", 2024-04-16T07:42:55.184 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_op_history_size": "20", 2024-04-16T07:42:55.184 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_op_history_slow_op_size": "20", 2024-04-16T07:42:55.185 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-16T07:42:55.185 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_op_log_threshold": "5", 2024-04-16T07:42:55.185 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-16T07:42:55.185 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-16T07:42:55.185 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-16T07:42:55.185 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_auto_mark_in": "false", 2024-04-16T07:42:55.185 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-16T07:42:55.185 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-16T07:42:55.185 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-16T07:42:55.185 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_cache_size": "500", 2024-04-16T07:42:55.185 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-16T07:42:55.185 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-16T07:42:55.185 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-16T07:42:55.185 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_down_out_interval": "600", 2024-04-16T07:42:55.185 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-16T07:42:55.185 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-16T07:42:55.186 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_force_trim_to": "0", 2024-04-16T07:42:55.186 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-16T07:42:55.186 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-16T07:42:55.186 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-16T07:42:55.186 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-16T07:42:55.186 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-16T07:42:55.186 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-16T07:42:55.186 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-16T07:42:55.186 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_min_down_reporters": "2", 2024-04-16T07:42:55.186 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-16T07:42:55.186 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-16T07:42:55.186 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-16T07:42:55.186 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-16T07:42:55.187 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-16T07:42:55.187 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-16T07:42:55.187 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_report_timeout": "900", 2024-04-16T07:42:55.187 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-16T07:42:55.187 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-16T07:42:55.187 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-16T07:42:55.187 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-16T07:42:55.187 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-16T07:42:55.187 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-16T07:42:55.187 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-16T07:42:55.187 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-16T07:42:55.187 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-16T07:42:55.187 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pg_stuck_threshold": "60", 2024-04-16T07:42:55.187 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-16T07:42:55.188 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-16T07:42:55.188 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-16T07:42:55.188 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-16T07:42:55.188 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-16T07:42:55.188 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-16T07:42:55.188 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_probe_timeout": "2.000000", 2024-04-16T07:42:55.188 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_reweight_max_change": "0.050000", 2024-04-16T07:42:55.188 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_reweight_max_osds": "4", 2024-04-16T07:42:55.188 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-16T07:42:55.188 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-16T07:42:55.188 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-16T07:42:55.188 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-16T07:42:55.188 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-16T07:42:55.188 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_scrub_interval": "86400", 2024-04-16T07:42:55.189 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_scrub_max_keys": "100", 2024-04-16T07:42:55.189 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_scrub_timeout": "300", 2024-04-16T07:42:55.189 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_session_timeout": "300", 2024-04-16T07:42:55.189 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_smart_report_timeout": "5", 2024-04-16T07:42:55.189 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_stat_smooth_intervals": "6", 2024-04-16T07:42:55.189 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-16T07:42:55.189 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_stretch_pool_min_size": "2", 2024-04-16T07:42:55.189 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_stretch_pool_size": "4", 2024-04-16T07:42:55.189 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-16T07:42:55.189 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-16T07:42:55.189 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_sync_debug": "false", 2024-04-16T07:42:55.189 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-16T07:42:55.189 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-16T07:42:55.189 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_sync_provider_kill_at": "0", 2024-04-16T07:42:55.189 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_sync_requester_kill_at": "0", 2024-04-16T07:42:55.190 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_sync_timeout": "60.000000", 2024-04-16T07:42:55.190 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_target_pg_per_osd": "100", 2024-04-16T07:42:55.190 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_tick_interval": "5", 2024-04-16T07:42:55.190 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_timecheck_interval": "300.000000", 2024-04-16T07:42:55.190 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-16T07:42:55.190 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_use_min_delay_socket": "false", 2024-04-16T07:42:55.190 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_older_version_delay": "604800", 2024-04-16T07:42:55.190 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-16T07:42:55.190 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-16T07:42:55.190 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-16T07:42:55.190 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-16T07:42:55.190 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-16T07:42:55.190 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-16T07:42:55.191 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-16T07:42:55.191 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_misplaced": "false", 2024-04-16T07:42:55.191 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-16T07:42:55.191 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_older_version": "true", 2024-04-16T07:42:55.191 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-16T07:42:55.191 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-16T07:42:55.191 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-16T07:42:55.191 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-16T07:42:55.191 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-16T07:42:55.191 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-16T07:42:55.191 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-16T07:42:55.191 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-16T07:42:55.191 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-16T07:42:55.191 INFO:teuthology.orchestra.run.smithi031.stdout: "monmap": "", 2024-04-16T07:42:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-16T07:42:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-16T07:42:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-16T07:42:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-16T07:42:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-16T07:42:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-16T07:42:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_tracing_enabled": "false", 2024-04-16T07:42:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_op_threads": "3", 2024-04-16T07:42:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-16T07:42:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_cm": "false", 2024-04-16T07:42:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_device_name": "", 2024-04-16T07:42:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_dscp": "96", 2024-04-16T07:42:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-16T07:42:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-16T07:42:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_local_gid": "", 2024-04-16T07:42:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-16T07:42:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_port_num": "1", 2024-04-16T07:42:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-16T07:42:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-16T07:42:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-16T07:42:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-16T07:42:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_sl": "3", 2024-04-16T07:42:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_support_srq": "true", 2024-04-16T07:42:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_type": "ib", 2024-04-16T07:42:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_reap_threshold": "5", 2024-04-16T07:42:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_before_connect": "false", 2024-04-16T07:42:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_ipv4": "true", 2024-04-16T07:42:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_ipv6": "false", 2024-04-16T07:42:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_msgr1": "true", 2024-04-16T07:42:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_msgr2": "true", 2024-04-16T07:42:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_port_max": "7568", 2024-04-16T07:42:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_port_min": "6800", 2024-04-16T07:42:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-16T07:42:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_retry_count": "3", 2024-04-16T07:42:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_retry_delay": "5", 2024-04-16T07:42:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_blackhole_client": "false", 2024-04-16T07:42:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_blackhole_mds": "false", 2024-04-16T07:42:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_blackhole_mgr": "false", 2024-04-16T07:42:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_blackhole_mon": "false", 2024-04-16T07:42:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_blackhole_osd": "false", 2024-04-16T07:42:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_client_mode": "crc", 2024-04-16T07:42:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_cluster_mode": "crc", 2024-04-16T07:42:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_cluster_type": "", 2024-04-16T07:42:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_compress_secure": "false", 2024-04-16T07:42:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_connection_idle_timeout": "900", 2024-04-16T07:42:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_connection_ready_timeout": "10", 2024-04-16T07:42:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_crc_data": "true", 2024-04-16T07:42:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_crc_header": "true", 2024-04-16T07:42:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_die_on_bad_msg": "false", 2024-04-16T07:42:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_die_on_bug": "true", 2024-04-16T07:42:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_die_on_old_message": "true", 2024-04-16T07:42:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_die_on_skipped_message": "false", 2024-04-16T07:42:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-16T07:42:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-16T07:42:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_coremask": "0xF", 2024-04-16T07:42:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-16T07:42:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-16T07:42:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_enable_tso": "true", 2024-04-16T07:42:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-16T07:42:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-16T07:42:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_hugepages": "", 2024-04-16T07:42:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-16T07:42:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-16T07:42:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_lro": "true", 2024-04-16T07:42:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_memory_channel": "4", 2024-04-16T07:42:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-16T07:42:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_pmd": "", 2024-04-16T07:42:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_port_id": "0", 2024-04-16T07:42:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-16T07:42:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-16T07:42:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dump_on_send": "false", 2024-04-16T07:42:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_initial_backoff": "0.200000", 2024-04-16T07:42:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_inject_delay_max": "1.000000", 2024-04-16T07:42:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-16T07:42:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_inject_delay_type": "", 2024-04-16T07:42:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-16T07:42:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_inject_network_congestion": "0", 2024-04-16T07:42:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_inject_socket_failures": "0", 2024-04-16T07:42:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_learn_addr_from_peer": "true", 2024-04-16T07:42:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_max_accept_failures": "4", 2024-04-16T07:42:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_max_backoff": "15.000000", 2024-04-16T07:42:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_mon_client_mode": "crc", 2024-04-16T07:42:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_mon_cluster_mode": "crc", 2024-04-16T07:42:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_mon_service_mode": "crc", 2024-04-16T07:42:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_osd_compress_min_size": "1024", 2024-04-16T07:42:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_osd_compress_mode": "none", 2024-04-16T07:42:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-16T07:42:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-16T07:42:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_pq_min_cost": "65536", 2024-04-16T07:42:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_public_type": "", 2024-04-16T07:42:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_service_mode": "crc", 2024-04-16T07:42:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_tcp_listen_backlog": "512", 2024-04-16T07:42:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_tcp_nodelay": "true", 2024-04-16T07:42:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-16T07:42:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_tcp_rcvbuf": "0", 2024-04-16T07:42:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_type": "async+posix", 2024-04-16T07:42:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "no_config_file": "false", 2024-04-16T07:42:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_completion_locks_per_session": "32", 2024-04-16T07:42:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-16T07:42:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-16T07:42:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_inflight_ops": "1024", 2024-04-16T07:42:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-16T07:42:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-16T07:42:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_tick_interval": "5.000000", 2024-04-16T07:42:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_timeout": "10.000000", 2024-04-16T07:42:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "objectstore_blackhole": "false", 2024-04-16T07:42:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-16T07:42:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "openssl_engine_opts": "", 2024-04-16T07:42:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_delay_time": "5.000000", 2024-04-16T07:42:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_hist_halflife": "1000", 2024-04-16T07:42:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_max_low_ops": "2", 2024-04-16T07:42:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_max_ops": "4", 2024-04-16T07:42:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-16T07:42:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-16T07:42:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_slop": "0.020000", 2024-04-16T07:42:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-16T07:42:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-16T07:42:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_asio_thread_count": "2", 2024-04-16T07:42:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_async_recovery_min_cost": "100", 2024-04-16T07:42:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-16T07:42:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-16T07:42:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_backfill_scan_max": "512", 2024-04-16T07:42:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_backfill_scan_min": "64", 2024-04-16T07:42:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_backoff_on_degraded": "false", 2024-04-16T07:42:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_backoff_on_peering": "false", 2024-04-16T07:42:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_backoff_on_unfound": "true", 2024-04-16T07:42:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_beacon_report_interval": "300", 2024-04-16T07:42:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bench_duration": "30", 2024-04-16T07:42:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-16T07:42:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bench_max_block_size": "67108864", 2024-04-16T07:42:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-16T07:42:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_blkin_trace_all": "false", 2024-04-16T07:42:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-16T07:42:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-16T07:42:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-16T07:42:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-16T07:42:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_check_for_log_corruption": "false", 2024-04-16T07:42:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-16T07:42:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-16T07:42:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-16T07:42:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-16T07:42:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_class_update_on_start": "true", 2024-04-16T07:42:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_client_message_cap": "256", 2024-04-16T07:42:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_client_message_size_cap": "524288000", 2024-04-16T07:42:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_client_op_priority": "63", 2024-04-16T07:42:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_client_watch_timeout": "30", 2024-04-16T07:42:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_command_max_records": "256", 2024-04-16T07:42:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_compact_on_start": "false", 2024-04-16T07:42:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-16T07:42:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-16T07:42:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-16T07:42:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_crush_update_on_start": "true", 2024-04-16T07:42:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_crush_update_weight_set": "true", 2024-04-16T07:42:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-16T07:42:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-16T07:42:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-16T07:42:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-16T07:42:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-16T07:42:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_feed_pullee": "-1", 2024-04-16T07:42:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-16T07:42:55.203 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-16T07:42:55.203 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-16T07:42:55.203 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_misdirected_ops": "false", 2024-04-16T07:42:55.203 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_no_acting_change": "false", 2024-04-16T07:42:55.203 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_no_purge_strays": "false", 2024-04-16T07:42:55.203 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_op_order": "false", 2024-04-16T07:42:55.203 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-16T07:42:55.203 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-16T07:42:55.203 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-16T07:42:55.203 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-16T07:42:55.203 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_shutdown": "false", 2024-04-16T07:42:55.203 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-16T07:42:55.203 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-16T07:42:55.203 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_trim_objects": "false", 2024-04-16T07:42:55.204 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-16T07:42:55.204 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-16T07:42:55.204 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_verify_snaps": "false", 2024-04-16T07:42:55.204 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-16T07:42:55.204 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-16T07:42:55.204 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_keys": "1024", 2024-04-16T07:42:55.204 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-16T07:42:55.204 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-16T07:42:55.204 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-16T07:42:55.204 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_stride": "524288", 2024-04-16T07:42:55.204 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-16T07:42:55.204 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-16T07:42:55.204 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_default_notify_timeout": "30", 2024-04-16T07:42:55.204 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_delete_sleep": "0.000000", 2024-04-16T07:42:55.205 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-16T07:42:55.205 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-16T07:42:55.205 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-16T07:42:55.205 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_discard_disconnected_ops": "true", 2024-04-16T07:42:55.205 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_enable_op_tracker": "true", 2024-04-16T07:42:55.205 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-16T07:42:55.205 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-16T07:42:55.205 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-16T07:42:55.205 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-16T07:42:55.205 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_fast_info": "true", 2024-04-16T07:42:55.205 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_fast_shutdown": "true", 2024-04-16T07:42:55.205 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-16T07:42:55.205 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-16T07:42:55.205 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-16T07:42:55.205 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-16T07:42:55.206 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-16T07:42:55.206 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_function_tracing": "false", 2024-04-16T07:42:55.206 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_grace": "20", 2024-04-16T07:42:55.206 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_interval": "6", 2024-04-16T07:42:55.206 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-16T07:42:55.206 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_min_peers": "10", 2024-04-16T07:42:55.206 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_min_size": "2000", 2024-04-16T07:42:55.206 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_stale": "600", 2024-04-16T07:42:55.206 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-16T07:42:55.206 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_hit_set_max_size": "100000", 2024-04-16T07:42:55.206 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_hit_set_min_size": "1000", 2024-04-16T07:42:55.206 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-16T07:42:55.206 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-16T07:42:55.207 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-16T07:42:55.207 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-16T07:42:55.207 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-16T07:42:55.207 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-16T07:42:55.207 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_journal_size": "5120", 2024-04-16T07:42:55.207 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_kill_backfill_at": "0", 2024-04-16T07:42:55.207 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-16T07:42:55.207 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_map_cache_size": "50", 2024-04-16T07:42:55.207 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_map_dedup": "true", 2024-04-16T07:42:55.207 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_map_message_max": "40", 2024-04-16T07:42:55.207 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-16T07:42:55.207 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_map_share_max_epochs": "40", 2024-04-16T07:42:55.207 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_attr_name_len": "100", 2024-04-16T07:42:55.207 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_attr_size": "0", 2024-04-16T07:42:55.208 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_backfills": "1", 2024-04-16T07:42:55.208 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_markdown_count": "5", 2024-04-16T07:42:55.208 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_markdown_period": "600", 2024-04-16T07:42:55.208 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_object_name_len": "2048", 2024-04-16T07:42:55.208 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_object_namespace_len": "256", 2024-04-16T07:42:55.208 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_object_size": "134217728", 2024-04-16T07:42:55.208 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-16T07:42:55.208 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-16T07:42:55.208 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_pg_blocked_by": "16", 2024-04-16T07:42:55.208 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_pg_log_entries": "10000", 2024-04-16T07:42:55.208 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-16T07:42:55.208 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_pgls": "1024", 2024-04-16T07:42:55.208 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_push_cost": "8388608", 2024-04-16T07:42:55.209 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_push_objects": "10", 2024-04-16T07:42:55.209 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_scrubs": "3", 2024-04-16T07:42:55.209 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-16T07:42:55.209 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_trimming_pgs": "2", 2024-04-16T07:42:55.209 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_write_op_reply_len": "64", 2024-04-16T07:42:55.209 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_write_size": "90", 2024-04-16T07:42:55.209 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-16T07:42:55.209 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-16T07:42:55.209 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-16T07:42:55.209 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-16T07:42:55.209 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-16T07:42:55.209 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-16T07:42:55.209 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-16T07:42:55.210 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-16T07:42:55.210 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_profile": "balanced", 2024-04-16T07:42:55.210 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-16T07:42:55.210 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-16T07:42:55.210 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-16T07:42:55.210 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-16T07:42:55.210 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-16T07:42:55.210 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-16T07:42:55.210 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-16T07:42:55.210 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-16T07:42:55.210 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-16T07:42:55.210 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-16T07:42:55.210 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-16T07:42:55.210 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_base": "805306368", 2024-04-16T07:42:55.211 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_cache_min": "134217728", 2024-04-16T07:42:55.211 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-16T07:42:55.211 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-16T07:42:55.211 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_target": "4294967296", 2024-04-16T07:42:55.211 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_target_autotune": "false", 2024-04-16T07:42:55.211 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-16T07:42:55.211 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_min_pg_log_entries": "250", 2024-04-16T07:42:55.211 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_min_recovery_priority": "0", 2024-04-16T07:42:55.211 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-16T07:42:55.211 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-16T07:42:55.211 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mon_report_interval": "5", 2024-04-16T07:42:55.211 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-16T07:42:55.211 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-16T07:42:55.211 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_num_cache_shards": "32", 2024-04-16T07:42:55.212 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_num_op_tracker_shard": "32", 2024-04-16T07:42:55.212 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_numa_auto_affinity": "true", 2024-04-16T07:42:55.212 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_numa_node": "-1", 2024-04-16T07:42:55.212 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_numa_prefer_iface": "true", 2024-04-16T07:42:55.212 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-16T07:42:55.212 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_objecter_finishers": "1", 2024-04-16T07:42:55.212 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-16T07:42:55.212 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_objectstore_fuse": "false", 2024-04-16T07:42:55.212 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_objectstore_tracing": "false", 2024-04-16T07:42:55.212 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_complaint_time": "30.000000", 2024-04-16T07:42:55.212 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_history_duration": "600", 2024-04-16T07:42:55.212 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_history_size": "20", 2024-04-16T07:42:55.212 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_history_slow_op_size": "20", 2024-04-16T07:42:55.212 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-16T07:42:55.213 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_log_threshold": "5", 2024-04-16T07:42:55.213 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_num_shards": "0", 2024-04-16T07:42:55.213 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_num_shards_hdd": "5", 2024-04-16T07:42:55.213 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_num_shards_ssd": "8", 2024-04-16T07:42:55.213 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-16T07:42:55.213 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-16T07:42:55.213 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-16T07:42:55.213 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-16T07:42:55.213 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_pq_min_cost": "65536", 2024-04-16T07:42:55.213 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-16T07:42:55.213 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_queue_cut_off": "high", 2024-04-16T07:42:55.213 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-16T07:42:55.213 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_thread_timeout": "15", 2024-04-16T07:42:55.213 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_open_classes_on_start": "true", 2024-04-16T07:42:55.214 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_os_flags": "0", 2024-04-16T07:42:55.214 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_peering_op_priority": "255", 2024-04-16T07:42:55.214 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_delete_cost": "1048576", 2024-04-16T07:42:55.214 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_delete_priority": "5", 2024-04-16T07:42:55.214 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-16T07:42:55.214 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-16T07:42:55.214 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-16T07:42:55.214 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_log_trim_max": "10000", 2024-04-16T07:42:55.214 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_log_trim_min": "100", 2024-04-16T07:42:55.214 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-16T07:42:55.214 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-16T07:42:55.214 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-16T07:42:55.214 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-16T07:42:55.215 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-16T07:42:55.215 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-16T07:42:55.215 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-16T07:42:55.215 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-16T07:42:55.215 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-16T07:42:55.215 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-16T07:42:55.215 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_crimson": "true", 2024-04-16T07:42:55.215 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-16T07:42:55.215 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-16T07:42:55.215 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-16T07:42:55.215 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-16T07:42:55.215 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-16T07:42:55.215 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-16T07:42:55.215 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-16T07:42:55.216 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-16T07:42:55.216 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_flags": "0", 2024-04-16T07:42:55.216 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-16T07:42:55.216 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_min_size": "2", 2024-04-16T07:42:55.216 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-16T07:42:55.216 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_pg_num": "32", 2024-04-16T07:42:55.216 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_pgp_num": "0", 2024-04-16T07:42:55.216 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-16T07:42:55.216 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_read_ratio": "70", 2024-04-16T07:42:55.216 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_size": "3", 2024-04-16T07:42:55.216 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_type": "replicated", 2024-04-16T07:42:55.216 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-16T07:42:55.216 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-16T07:42:55.216 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_push_per_object_cost": "1000", 2024-04-16T07:42:55.217 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-16T07:42:55.217 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recover_clone_overlap": "true", 2024-04-16T07:42:55.217 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-16T07:42:55.217 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_cost": "20971520", 2024-04-16T07:42:55.217 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-16T07:42:55.217 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_max_active": "0", 2024-04-16T07:42:55.217 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-16T07:42:55.217 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-16T07:42:55.217 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-16T07:42:55.217 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-16T07:42:55.217 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_max_single_start": "1", 2024-04-16T07:42:55.217 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_op_priority": "3", 2024-04-16T07:42:55.217 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-16T07:42:55.217 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_priority": "5", 2024-04-16T07:42:55.217 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-16T07:42:55.218 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_sleep": "0.000000", 2024-04-16T07:42:55.218 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-16T07:42:55.218 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-16T07:42:55.218 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-16T07:42:55.218 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_repair_during_recovery": "false", 2024-04-16T07:42:55.218 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_requested_scrub_priority": "120", 2024-04-16T07:42:55.218 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-16T07:42:55.218 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-16T07:42:55.218 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_auto_repair": "false", 2024-04-16T07:42:55.218 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-16T07:42:55.218 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-16T07:42:55.218 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_begin_hour": "0", 2024-04-16T07:42:55.218 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_begin_week_day": "0", 2024-04-16T07:42:55.219 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_chunk_max": "25", 2024-04-16T07:42:55.219 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_chunk_min": "5", 2024-04-16T07:42:55.219 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_cost": "52428800", 2024-04-16T07:42:55.219 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-16T07:42:55.219 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_during_recovery": "false", 2024-04-16T07:42:55.219 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_end_hour": "0", 2024-04-16T07:42:55.219 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_end_week_day": "0", 2024-04-16T07:42:55.219 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_event_cost": "4096", 2024-04-16T07:42:55.219 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-16T07:42:55.219 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-16T07:42:55.219 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_invalid_stats": "true", 2024-04-16T07:42:55.219 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-16T07:42:55.219 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-16T07:42:55.219 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_max_preemptions": "5", 2024-04-16T07:42:55.219 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-16T07:42:55.220 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_priority": "5", 2024-04-16T07:42:55.220 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-16T07:42:55.220 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_sleep": "0.000000", 2024-04-16T07:42:55.220 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-16T07:42:55.220 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-16T07:42:55.220 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-16T07:42:55.220 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-04-16T07:42:55.220 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_skip_data_digest": "false", 2024-04-16T07:42:55.220 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_smart_report_timeout": "5", 2024-04-16T07:42:55.220 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_snap_trim_cost": "1048576", 2024-04-16T07:42:55.220 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_snap_trim_priority": "5", 2024-04-16T07:42:55.220 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-16T07:42:55.220 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-16T07:42:55.221 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-16T07:42:55.221 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-16T07:42:55.221 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-16T07:42:55.221 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-16T07:42:55.221 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-16T07:42:55.221 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_target_transaction_size": "30", 2024-04-16T07:42:55.221 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-16T07:42:55.221 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-16T07:42:55.221 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-16T07:42:55.221 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-16T07:42:55.221 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-16T07:42:55.221 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-16T07:42:55.221 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-16T07:42:55.221 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-16T07:42:55.222 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-16T07:42:55.222 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-16T07:42:55.222 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tracing": "false", 2024-04-16T07:42:55.222 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_use_stale_snap": "false", 2024-04-16T07:42:55.222 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-16T07:42:55.222 INFO:teuthology.orchestra.run.smithi031.stdout: "osdc_blkin_trace_all": "false", 2024-04-16T07:42:55.222 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_kill_at": "0", 2024-04-16T07:42:55.222 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_max_join_drift": "10", 2024-04-16T07:42:55.222 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_min": "500", 2024-04-16T07:42:55.222 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_min_wait": "0.050000", 2024-04-16T07:42:55.222 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_propose_interval": "1.000000", 2024-04-16T07:42:55.222 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_service_trim_max": "500", 2024-04-16T07:42:55.222 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-16T07:42:55.222 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_service_trim_min": "10", 2024-04-16T07:42:55.223 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_stash_full_interval": "25", 2024-04-16T07:42:55.223 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_trim_max": "500", 2024-04-16T07:42:55.223 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_trim_min": "250", 2024-04-16T07:42:55.223 INFO:teuthology.orchestra.run.smithi031.stdout: "perf": "true", 2024-04-16T07:42:55.223 INFO:teuthology.orchestra.run.smithi031.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-16T07:42:55.223 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-16T07:42:55.223 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-16T07:42:55.223 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "-", 2024-04-16T07:42:55.223 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrv": "", 2024-04-16T07:42:55.223 INFO:teuthology.orchestra.run.smithi031.stdout: "public_bind_addr": "-", 2024-04-16T07:42:55.223 INFO:teuthology.orchestra.run.smithi031.stdout: "public_network": "", 2024-04-16T07:42:55.223 INFO:teuthology.orchestra.run.smithi031.stdout: "public_network_interface": "", 2024-04-16T07:42:55.223 INFO:teuthology.orchestra.run.smithi031.stdout: "qat_compressor_enabled": "false", 2024-04-16T07:42:55.223 INFO:teuthology.orchestra.run.smithi031.stdout: "qat_compressor_session_max_number": "256", 2024-04-16T07:42:55.224 INFO:teuthology.orchestra.run.smithi031.stdout: "rados_mon_op_timeout": "0", 2024-04-16T07:42:55.224 INFO:teuthology.orchestra.run.smithi031.stdout: "rados_osd_op_timeout": "0", 2024-04-16T07:42:55.224 INFO:teuthology.orchestra.run.smithi031.stdout: "rados_tracing": "false", 2024-04-16T07:42:55.224 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_atime_update_interval": "60", 2024-04-16T07:42:55.224 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-16T07:42:55.224 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_balance_parent_reads": "false", 2024-04-16T07:42:55.224 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_balance_snap_reads": "false", 2024-04-16T07:42:55.224 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_blkin_trace_all": "false", 2024-04-16T07:42:55.224 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-16T07:42:55.224 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-16T07:42:55.224 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache": "true", 2024-04-16T07:42:55.224 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-16T07:42:55.224 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-16T07:42:55.224 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-16T07:42:55.225 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-16T07:42:55.225 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_policy": "writearound", 2024-04-16T07:42:55.225 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_size": "33554432", 2024-04-16T07:42:55.225 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-16T07:42:55.225 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-16T07:42:55.225 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_clone_copy_on_read": "false", 2024-04-16T07:42:55.225 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_compression_hint": "none", 2024-04-16T07:42:55.225 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_concurrent_management_ops": "10", 2024-04-16T07:42:55.225 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-16T07:42:55.225 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_clone_format": "auto", 2024-04-16T07:42:55.225 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_data_pool": "", 2024-04-16T07:42:55.225 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_features": "61", 2024-04-16T07:42:55.225 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_format": "2", 2024-04-16T07:42:55.225 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_map_options": "", 2024-04-16T07:42:55.226 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_order": "22", 2024-04-16T07:42:55.226 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_pool": "rbd", 2024-04-16T07:42:55.226 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-16T07:42:55.226 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_stripe_count": "0", 2024-04-16T07:42:55.226 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_stripe_unit": "0", 2024-04-16T07:42:55.226 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-16T07:42:55.226 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-16T07:42:55.226 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-16T07:42:55.226 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_enable_alloc_hint": "true", 2024-04-16T07:42:55.226 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-16T07:42:55.226 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_io_scheduler": "simple", 2024-04-16T07:42:55.226 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-16T07:42:55.226 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-16T07:42:55.226 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-16T07:42:55.227 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-16T07:42:55.227 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-16T07:42:55.227 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-16T07:42:55.227 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-16T07:42:55.227 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-16T07:42:55.227 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-16T07:42:55.227 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_order": "24", 2024-04-16T07:42:55.227 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_pool": "", 2024-04-16T07:42:55.227 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_splay_width": "4", 2024-04-16T07:42:55.227 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_localize_parent_reads": "false", 2024-04-16T07:42:55.227 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_localize_snap_reads": "false", 2024-04-16T07:42:55.227 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-16T07:42:55.227 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-16T07:42:55.227 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-16T07:42:55.228 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-16T07:42:55.228 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-16T07:42:55.228 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-16T07:42:55.228 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-16T07:42:55.228 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-16T07:42:55.228 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-16T07:42:55.228 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-16T07:42:55.228 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-16T07:42:55.228 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-16T07:42:55.228 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-16T07:42:55.228 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-16T07:42:55.228 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-16T07:42:55.228 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-16T07:42:55.228 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-16T07:42:55.229 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-16T07:42:55.229 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-16T07:42:55.229 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-16T07:42:55.229 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-16T07:42:55.229 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-16T07:42:55.229 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-16T07:42:55.229 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-16T07:42:55.229 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-16T07:42:55.229 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-16T07:42:55.229 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-16T07:42:55.229 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-16T07:42:55.229 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-16T07:42:55.229 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-16T07:42:55.229 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-16T07:42:55.230 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mtime_update_interval": "60", 2024-04-16T07:42:55.230 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_non_blocking_aio": "true", 2024-04-16T07:42:55.230 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_op_thread_timeout": "60", 2024-04-16T07:42:55.230 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_op_threads": "1", 2024-04-16T07:42:55.230 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_parent_cache_enabled": "false", 2024-04-16T07:42:55.230 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-16T07:42:55.230 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-16T07:42:55.230 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-16T07:42:55.230 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_plugins": "", 2024-04-16T07:42:55.230 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_bps_burst": "0", 2024-04-16T07:42:55.230 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-16T07:42:55.230 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_bps_limit": "0", 2024-04-16T07:42:55.230 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_exclude_ops": "0", 2024-04-16T07:42:55.231 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_iops_burst": "0", 2024-04-16T07:42:55.231 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-16T07:42:55.231 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_iops_limit": "0", 2024-04-16T07:42:55.231 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-16T07:42:55.231 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-16T07:42:55.231 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-16T07:42:55.231 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-16T07:42:55.231 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-16T07:42:55.231 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-16T07:42:55.231 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-16T07:42:55.231 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-16T07:42:55.231 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-16T07:42:55.231 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-16T07:42:55.231 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-16T07:42:55.231 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-16T07:42:55.232 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-16T07:42:55.232 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-16T07:42:55.232 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_read_from_replica_policy": "default", 2024-04-16T07:42:55.232 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-16T07:42:55.232 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-16T07:42:55.232 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-16T07:42:55.232 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-16T07:42:55.232 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_skip_partial_discard": "true", 2024-04-16T07:42:55.232 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-16T07:42:55.232 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_tracing": "false", 2024-04-16T07:42:55.232 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_validate_names": "true", 2024-04-16T07:42:55.232 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_validate_pool": "true", 2024-04-16T07:42:55.232 INFO:teuthology.orchestra.run.smithi031.stdout: "restapi_base_url": "", 2024-04-16T07:42:55.232 INFO:teuthology.orchestra.run.smithi031.stdout: "restapi_log_level": "", 2024-04-16T07:42:55.233 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_acl_grants_max_num": "100", 2024-04-16T07:42:55.233 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_admin_entry": "admin", 2024-04-16T07:42:55.233 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-16T07:42:55.233 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_backend_store": "rados", 2024-04-16T07:42:55.233 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_barbican_url": "", 2024-04-16T07:42:55.233 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_beast_enable_async": "true", 2024-04-16T07:42:55.233 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_counters_cache": "false", 2024-04-16T07:42:55.233 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-16T07:42:55.233 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-16T07:42:55.233 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-16T07:42:55.233 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-16T07:42:55.233 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-16T07:42:55.233 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-16T07:42:55.233 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-16T07:42:55.233 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-16T07:42:55.234 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_cache_enabled": "true", 2024-04-16T07:42:55.234 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_cache_expiry_interval": "900", 2024-04-16T07:42:55.234 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_cache_lru_size": "10000", 2024-04-16T07:42:55.234 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_config_store": "rados", 2024-04-16T07:42:55.234 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_content_length_compat": "false", 2024-04-16T07:42:55.234 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_copy_obj_progress": "true", 2024-04-16T07:42:55.234 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-16T07:42:55.234 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_cors_rules_max_num": "100", 2024-04-16T07:42:55.234 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_cross_domain_policy": "", 2024-04-16T07:42:55.234 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-16T07:42:55.234 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_addr": "", 2024-04-16T07:42:55.234 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-16T07:42:55.234 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-16T07:42:55.235 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-16T07:42:55.235 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-16T07:42:55.235 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_password": "", 2024-04-16T07:42:55.235 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-16T07:42:55.235 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_username": "", 2024-04-16T07:42:55.235 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_require_ssl": "true", 2024-04-16T07:42:55.235 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-16T07:42:55.235 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-16T07:42:55.235 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-16T07:42:55.235 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-16T07:42:55.235 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-16T07:42:55.235 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-16T07:42:55.235 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-16T07:42:55.235 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-16T07:42:55.236 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-16T07:42:55.236 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-16T07:42:55.236 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-16T07:42:55.236 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-16T07:42:55.236 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-16T07:42:55.236 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-16T07:42:55.236 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-16T07:42:55.236 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_addr": "", 2024-04-16T07:42:55.236 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_auth": "token", 2024-04-16T07:42:55.236 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_namespace": "", 2024-04-16T07:42:55.236 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_prefix": "", 2024-04-16T07:42:55.236 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-16T07:42:55.236 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-16T07:42:55.236 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-16T07:42:55.237 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-16T07:42:55.237 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_token_file": "", 2024-04-16T07:42:55.237 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-16T07:42:55.237 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_curl_buffersize": "524288", 2024-04-16T07:42:55.237 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-16T07:42:55.237 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_curl_low_speed_time": "300", 2024-04-16T07:42:55.237 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-16T07:42:55.237 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-16T07:42:55.237 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-16T07:42:55.237 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-16T07:42:55.237 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-16T07:42:55.237 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-16T07:42:55.237 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-16T07:42:55.237 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-16T07:42:55.238 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-16T07:42:55.238 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-16T07:42:55.238 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-16T07:42:55.238 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-04-16T07:42:55.238 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-04-16T07:42:55.238 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-04-16T07:42:55.238 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d4n_l1_fadvise": "4", 2024-04-16T07:42:55.238 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-04-16T07:42:55.238 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-04-16T07:42:55.238 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d4n_port": "6379", 2024-04-16T07:42:55.238 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-16T07:42:55.238 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_log_changes_size": "1000", 2024-04-16T07:42:55.238 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_log_num_shards": "128", 2024-04-16T07:42:55.238 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-16T07:42:55.239 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_log_window": "30", 2024-04-16T07:42:55.239 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-16T07:42:55.239 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-16T07:42:55.239 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-16T07:42:55.239 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-16T07:42:55.239 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-16T07:42:55.239 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-16T07:42:55.239 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-16T07:42:55.239 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-16T07:42:55.239 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-16T07:42:55.239 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-16T07:42:55.239 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-16T07:42:55.239 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-16T07:42:55.239 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-16T07:42:55.240 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-16T07:42:55.240 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-16T07:42:55.240 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-16T07:42:55.240 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-16T07:42:55.240 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-16T07:42:55.240 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-16T07:42:55.240 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-16T07:42:55.240 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-16T07:42:55.240 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-16T07:42:55.240 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-16T07:42:55.240 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-16T07:42:55.240 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-16T07:42:55.240 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dns_name": "", 2024-04-16T07:42:55.240 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dns_s3website_name": "", 2024-04-16T07:42:55.240 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dynamic_resharding": "true", 2024-04-16T07:42:55.241 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-16T07:42:55.241 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_gc_threads": "true", 2024-04-16T07:42:55.241 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_lc_threads": "true", 2024-04-16T07:42:55.241 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_ops_log": "false", 2024-04-16T07:42:55.241 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_quota_threads": "true", 2024-04-16T07:42:55.241 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_static_website": "false", 2024-04-16T07:42:55.241 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_usage_log": "false", 2024-04-16T07:42:55.241 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enforce_swift_acls": "true", 2024-04-16T07:42:55.241 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_exit_timeout_secs": "120", 2024-04-16T07:42:55.241 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_expose_bucket": "false", 2024-04-16T07:42:55.241 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_extended_http_attrs": "", 2024-04-16T07:42:55.241 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_filter": "none", 2024-04-16T07:42:55.241 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-16T07:42:55.241 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_frontends": "beast port=7480", 2024-04-16T07:42:55.241 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-16T07:42:55.242 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_max_deferred": "50", 2024-04-16T07:42:55.242 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-16T07:42:55.242 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_max_objs": "32", 2024-04-16T07:42:55.242 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-16T07:42:55.242 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-16T07:42:55.242 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-16T07:42:55.242 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-16T07:42:55.242 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_processor_period": "3600", 2024-04-16T07:42:55.242 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-16T07:42:55.242 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-16T07:42:55.242 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-16T07:42:55.242 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-16T07:42:55.242 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_init_timeout": "300", 2024-04-16T07:42:55.243 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-16T07:42:55.243 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-16T07:42:55.243 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_kafka_connection_idle": "300", 2024-04-16T07:42:55.243 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-16T07:42:55.243 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-16T07:42:55.243 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-16T07:42:55.243 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-16T07:42:55.243 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-16T07:42:55.243 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_domain": "", 2024-04-16T07:42:55.243 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_password": "", 2024-04-16T07:42:55.243 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_password_path": "", 2024-04-16T07:42:55.243 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_project": "", 2024-04-16T07:42:55.243 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_tenant": "", 2024-04-16T07:42:55.243 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_token": "", 2024-04-16T07:42:55.243 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_token_path": "", 2024-04-16T07:42:55.243 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_user": "", 2024-04-16T07:42:55.244 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_api_version": "2", 2024-04-16T07:42:55.244 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_barbican_domain": "", 2024-04-16T07:42:55.244 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_barbican_password": "", 2024-04-16T07:42:55.244 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_barbican_project": "", 2024-04-16T07:42:55.244 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-16T07:42:55.244 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_barbican_user": "", 2024-04-16T07:42:55.244 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-16T07:42:55.244 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-16T07:42:55.244 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-16T07:42:55.244 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-16T07:42:55.244 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-16T07:42:55.244 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_url": "", 2024-04-16T07:42:55.244 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-16T07:42:55.245 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_debug_interval": "-1", 2024-04-16T07:42:55.245 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_lock_max_time": "90", 2024-04-16T07:42:55.245 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_max_objs": "32", 2024-04-16T07:42:55.245 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_max_rules": "1000", 2024-04-16T07:42:55.245 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_max_worker": "3", 2024-04-16T07:42:55.245 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-16T07:42:55.245 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_thread_delay": "0", 2024-04-16T07:42:55.245 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-16T07:42:55.245 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ldap_dnattr": "uid", 2024-04-16T07:42:55.245 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-16T07:42:55.245 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ldap_searchfilter": "", 2024-04-16T07:42:55.245 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-16T07:42:55.245 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ldap_uri": "", 2024-04-16T07:42:55.245 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lfuda_sync_frequency": "60", 2024-04-16T07:42:55.246 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-16T07:42:55.246 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-16T07:42:55.246 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-16T07:42:55.246 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_local_cache_address": "127.0.0.1:6379", 2024-04-16T07:42:55.246 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_log_http_headers": "", 2024-04-16T07:42:55.246 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-16T07:42:55.246 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-16T07:42:55.246 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_log_object_name_utc": "false", 2024-04-16T07:42:55.246 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-16T07:42:55.246 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-16T07:42:55.246 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_attr_name_len": "0", 2024-04-16T07:42:55.246 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_attr_size": "0", 2024-04-16T07:42:55.246 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-16T07:42:55.247 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_chunk_size": "4194304", 2024-04-16T07:42:55.247 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-16T07:42:55.247 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-16T07:42:55.247 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-16T07:42:55.247 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_listing_results": "1000", 2024-04-16T07:42:55.247 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_notify_retries": "10", 2024-04-16T07:42:55.247 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-16T07:42:55.247 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_put_param_size": "1048576", 2024-04-16T07:42:55.247 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_put_size": "5368709120", 2024-04-16T07:42:55.247 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_slo_entries": "1000", 2024-04-16T07:42:55.247 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_md_log_max_shards": "64", 2024-04-16T07:42:55.247 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-16T07:42:55.247 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-16T07:42:55.247 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-16T07:42:55.248 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-16T07:42:55.248 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_mp_lock_max_time": "600", 2024-04-16T07:42:55.248 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-16T07:42:55.248 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-16T07:42:55.248 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-16T07:42:55.248 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-16T07:42:55.248 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-16T07:42:55.248 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-16T07:42:55.248 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-16T07:42:55.248 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-16T07:42:55.248 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_max_gc": "300", 2024-04-16T07:42:55.248 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-16T07:42:55.248 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-16T07:42:55.248 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-16T07:42:55.248 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-16T07:42:55.249 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-16T07:42:55.249 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-16T07:42:55.249 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-16T07:42:55.249 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_num_async_rados_threads": "32", 2024-04-16T07:42:55.249 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_num_control_oids": "8", 2024-04-16T07:42:55.249 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_numa_node": "-1", 2024-04-16T07:42:55.249 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-16T07:42:55.249 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-16T07:42:55.249 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_objexp_chunk_size": "100", 2024-04-16T07:42:55.249 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_objexp_gc_interval": "600", 2024-04-16T07:42:55.249 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-16T07:42:55.249 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-16T07:42:55.249 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-16T07:42:55.249 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_op_thread_timeout": "600", 2024-04-16T07:42:55.249 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_op_tracing": "false", 2024-04-16T07:42:55.250 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_opa_token": "", 2024-04-16T07:42:55.250 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_opa_url": "", 2024-04-16T07:42:55.250 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_opa_verify_ssl": "true", 2024-04-16T07:42:55.250 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-16T07:42:55.250 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-16T07:42:55.250 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ops_log_rados": "false", 2024-04-16T07:42:55.250 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ops_log_socket_path": "", 2024-04-16T07:42:55.250 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-16T07:42:55.250 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-16T07:42:55.250 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-16T07:42:55.250 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_period_push_interval": "2.000000", 2024-04-16T07:42:55.250 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-16T07:42:55.250 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-16T07:42:55.250 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-16T07:42:55.251 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-16T07:42:55.251 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_posix_cache_lanes": "3", 2024-04-16T07:42:55.251 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-16T07:42:55.251 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-16T07:42:55.251 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_posix_cache_partitions": "3", 2024-04-16T07:42:55.251 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-16T07:42:55.251 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_print_continue": "true", 2024-04-16T07:42:55.251 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-16T07:42:55.251 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-16T07:42:55.251 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-16T07:42:55.251 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-16T07:42:55.251 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-16T07:42:55.251 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_rados_tracing": "false", 2024-04-16T07:42:55.251 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_realm": "", 2024-04-16T07:42:55.252 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_realm_id": "", 2024-04-16T07:42:55.252 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-16T07:42:55.252 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_region": "", 2024-04-16T07:42:55.252 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-16T07:42:55.252 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-16T07:42:55.252 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-16T07:42:55.252 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-16T07:42:55.252 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_request_uri": "", 2024-04-16T07:42:55.252 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_reshard_batch_size": "64", 2024-04-16T07:42:55.252 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-16T07:42:55.252 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_reshard_max_aio": "128", 2024-04-16T07:42:55.252 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_reshard_num_logs": "16", 2024-04-16T07:42:55.252 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_reshard_thread_interval": "600", 2024-04-16T07:42:55.252 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_resolve_cname": "false", 2024-04-16T07:42:55.253 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-16T07:42:55.253 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_run_sync_thread": "true", 2024-04-16T07:42:55.253 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-16T07:42:55.253 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-16T07:42:55.253 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-16T07:42:55.253 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-16T07:42:55.253 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-16T07:42:55.253 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-16T07:42:55.253 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-16T07:42:55.253 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-16T07:42:55.253 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-16T07:42:55.253 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_scheduler_type": "throttler", 2024-04-16T07:42:55.253 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_script_uri": "", 2024-04-16T07:42:55.253 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_service_provider_name": "", 2024-04-16T07:42:55.253 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-16T07:42:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_client_id": "", 2024-04-16T07:42:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_client_secret": "", 2024-04-16T07:42:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_entry": "sts", 2024-04-16T07:42:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_key": "sts", 2024-04-16T07:42:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-16T07:42:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_min_session_duration": "900", 2024-04-16T07:42:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_token_introspection_url": "", 2024-04-16T07:42:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_account_in_url": "false", 2024-04-16T07:42:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_auth_entry": "auth", 2024-04-16T07:42:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_auth_url": "", 2024-04-16T07:42:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_custom_header": "", 2024-04-16T07:42:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-16T07:42:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_need_stats": "true", 2024-04-16T07:42:55.255 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_tenant_name": "", 2024-04-16T07:42:55.255 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_token_expiration": "86400", 2024-04-16T07:42:55.255 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_url": "", 2024-04-16T07:42:55.255 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_url_prefix": "swift", 2024-04-16T07:42:55.255 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-16T07:42:55.255 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-16T07:42:55.255 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-16T07:42:55.255 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_lease_period": "120", 2024-04-16T07:42:55.255 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-16T07:42:55.255 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-16T07:42:55.255 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-16T07:42:55.255 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-16T07:42:55.255 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-16T07:42:55.256 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-16T07:42:55.256 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-16T07:42:55.256 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-16T07:42:55.256 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-16T07:42:55.256 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_thread_pool_size": "512", 2024-04-16T07:42:55.256 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-16T07:42:55.256 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-16T07:42:55.256 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-16T07:42:55.256 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-16T07:42:55.256 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_comment": "", 2024-04-16T07:42:55.256 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_createby": "", 2024-04-16T07:42:55.256 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_encoding": "", 2024-04-16T07:42:55.256 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_flag": "false", 2024-04-16T07:42:55.256 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-16T07:42:55.256 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_origin": "", 2024-04-16T07:42:55.257 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-16T07:42:55.257 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_tracker": "", 2024-04-16T07:42:55.257 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_trust_forwarded_https": "false", 2024-04-16T07:42:55.257 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-16T07:42:55.257 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-16T07:42:55.257 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_usage_max_shards": "32", 2024-04-16T07:42:55.257 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_usage_max_user_shards": "1", 2024-04-16T07:42:55.257 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_use_opa_authz": "false", 2024-04-16T07:42:55.257 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_counters_cache": "false", 2024-04-16T07:42:55.257 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-16T07:42:55.257 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-16T07:42:55.257 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-16T07:42:55.257 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_max_buckets": "1000", 2024-04-16T07:42:55.257 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_policies_max_num": "100", 2024-04-16T07:42:55.258 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-16T07:42:55.258 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-16T07:42:55.258 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-16T07:42:55.258 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-16T07:42:55.258 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_unique_email": "true", 2024-04-16T07:42:55.258 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_verify_ssl": "true", 2024-04-16T07:42:55.258 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-16T07:42:55.258 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_zone": "", 2024-04-16T07:42:55.258 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_zone_id": "", 2024-04-16T07:42:55.258 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-16T07:42:55.258 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_zonegroup": "", 2024-04-16T07:42:55.258 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_zonegroup_id": "", 2024-04-16T07:42:55.258 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-16T07:42:55.258 INFO:teuthology.orchestra.run.smithi031.stdout: "rgwlc_auto_session_clear": "true", 2024-04-16T07:42:55.259 INFO:teuthology.orchestra.run.smithi031.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-16T07:42:55.259 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_block_size": "4096", 2024-04-16T07:42:55.259 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-16T07:42:55.259 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-16T07:42:55.259 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-16T07:42:55.259 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-16T07:42:55.259 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-16T07:42:55.259 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cache_size": "536870912", 2024-04-16T07:42:55.259 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-16T07:42:55.259 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-16T07:42:55.259 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-16T07:42:55.259 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-16T07:42:55.259 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-16T07:42:55.259 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-16T07:42:55.260 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-16T07:42:55.260 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-16T07:42:55.260 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_index_type": "binary_search", 2024-04-16T07:42:55.260 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-16T07:42:55.260 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-16T07:42:55.260 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_partition_filters": "false", 2024-04-16T07:42:55.260 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_perf": "false", 2024-04-16T07:42:55.260 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-16T07:42:55.260 INFO:teuthology.orchestra.run.smithi031.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-16T07:42:55.260 INFO:teuthology.orchestra.run.smithi031.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-16T07:42:55.260 INFO:teuthology.orchestra.run.smithi031.stdout: "run_dir": "/var/run/ceph", 2024-04-16T07:42:55.260 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_block_create": "true", 2024-04-16T07:42:55.260 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_cache_lru_size": "67108864", 2024-04-16T07:42:55.260 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-16T07:42:55.261 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-16T07:42:55.261 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_default_max_object_size": "16777216", 2024-04-16T07:42:55.261 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-16T07:42:55.261 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_device_size": "53687091200", 2024-04-16T07:42:55.261 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_journal_batch_capacity": "16", 2024-04-16T07:42:55.261 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-16T07:42:55.261 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-16T07:42:55.261 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-16T07:42:55.261 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_main_device_type": "SSD", 2024-04-16T07:42:55.261 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-16T07:42:55.261 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-16T07:42:55.261 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-16T07:42:55.261 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-16T07:42:55.261 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-16T07:42:55.261 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_segment_size": "67108864", 2024-04-16T07:42:55.262 INFO:teuthology.orchestra.run.smithi031.stdout: "set_keepcaps": "false", 2024-04-16T07:42:55.262 INFO:teuthology.orchestra.run.smithi031.stdout: "setgroup": "", 2024-04-16T07:42:55.262 INFO:teuthology.orchestra.run.smithi031.stdout: "setuser": "", 2024-04-16T07:42:55.262 INFO:teuthology.orchestra.run.smithi031.stdout: "setuser_match_path": "", 2024-04-16T07:42:55.262 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-16T07:42:55.262 INFO:teuthology.orchestra.run.smithi031.stdout: "thp": "false", 2024-04-16T07:42:55.262 INFO:teuthology.orchestra.run.smithi031.stdout: "threadpool_default_timeout": "60", 2024-04-16T07:42:55.262 INFO:teuthology.orchestra.run.smithi031.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-16T07:42:55.262 INFO:teuthology.orchestra.run.smithi031.stdout: "throttler_perf_counter": "true" 2024-04-16T07:42:55.262 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-16T07:42:55.264 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-16T07:42:55.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:55.376+0000 7f65304ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:55.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:55.383+0000 7f65304ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:55.397 INFO:teuthology.orchestra.run.smithi031.stdout:{} 2024-04-16T07:42:55.397 INFO:teuthology.orchestra.run.smithi031.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-16T07:42:55.407 DEBUG:teuthology.orchestra.run.smithi031:> type ceph-objectstore-tool 2024-04-16T07:42:55.421 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-16T07:42:55.422 DEBUG:teuthology.orchestra.run.smithi094:> type ceph-objectstore-tool 2024-04-16T07:42:55.439 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-16T07:42:55.439 INFO:teuthology.run_tasks:Running task radosbench... 2024-04-16T07:42:55.449 INFO:tasks.radosbench:Beginning radosbench... 2024-04-16T07:42:55.449 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-16T07:42:55.449 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-04-16T07:42:55.450 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-16T07:42:55.450 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-16T07:42:55.450 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:42:55.450 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:42:55.451 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-16T07:42:55.451 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:42:55.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:55.579+0000 7f28dd42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:55.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:55.579+0000 7f9112d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:55.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:55.586+0000 7f28dd42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:55.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:55.579+0000 7feccc0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:55.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:55.588+0000 7f9112d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:55.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:55.594+0000 7feccc0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:55.799 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:42:55.800 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-16T07:42:55.812 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-04-16T07:42:55.218645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107952,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379408,"statfs":{"total":386547056640,"available":386436513792,"internally_reserved":0,"allocated":634880,"data_stored":1883310,"data_compressed":89856,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.965655"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.374760+0000","last_change":"2024-04-16T07:42:45.376121+0000","last_active":"2024-04-16T07:42:45.374760+0000","last_peered":"2024-04-16T07:42:45.374760+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.370390+0000","last_became_peered":"2024-04-16T07:42:45.370390+0000","last_unstale":"2024-04-16T07:42:45.374760+0000","last_undegraded":"2024-04-16T07:42:45.374760+0000","last_fullsized":"2024-04-16T07:42:45.374760+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.331165+0000","last_change":"2024-04-16T07:42:45.333004+0000","last_active":"2024-04-16T07:42:45.331165+0000","last_peered":"2024-04-16T07:42:45.331165+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.323737+0000","last_became_peered":"2024-04-16T07:42:45.323737+0000","last_unstale":"2024-04-16T07:42:45.331165+0000","last_undegraded":"2024-04-16T07:42:45.331165+0000","last_fullsized":"2024-04-16T07:42:45.331165+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:42:46.275422+0000","last_change":"2024-04-16T07:42:45.332152+0000","last_active":"2024-04-16T07:42:46.275422+0000","last_peered":"2024-04-16T07:42:46.275422+0000","last_clean":"2024-04-16T07:42:46.275422+0000","last_became_active":"2024-04-16T07:42:45.325833+0000","last_became_peered":"2024-04-16T07:42:45.325833+0000","last_unstale":"2024-04-16T07:42:46.275422+0000","last_undegraded":"2024-04-16T07:42:46.275422+0000","last_fullsized":"2024-04-16T07:42:46.275422+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.339473+0000","last_change":"2024-04-16T07:42:45.340605+0000","last_active":"2024-04-16T07:42:45.339473+0000","last_peered":"2024-04-16T07:42:45.339473+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.335238+0000","last_became_peered":"2024-04-16T07:42:45.335238+0000","last_unstale":"2024-04-16T07:42:45.339473+0000","last_undegraded":"2024-04-16T07:42:45.339473+0000","last_fullsized":"2024-04-16T07:42:45.339473+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.356397+0000","last_change":"2024-04-16T07:42:45.357843+0000","last_active":"2024-04-16T07:42:45.356397+0000","last_peered":"2024-04-16T07:42:45.356397+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.348908+0000","last_became_peered":"2024-04-16T07:42:45.348908+0000","last_unstale":"2024-04-16T07:42:45.356397+0000","last_undegraded":"2024-04-16T07:42:45.356397+0000","last_fullsized":"2024-04-16T07:42:45.356397+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.325226+0000","last_change":"2024-04-16T07:42:45.327681+0000","last_active":"2024-04-16T07:42:45.325226+0000","last_peered":"2024-04-16T07:42:45.325226+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.318385+0000","last_became_peered":"2024-04-16T07:42:45.318385+0000","last_unstale":"2024-04-16T07:42:45.325226+0000","last_undegraded":"2024-04-16T07:42:45.325226+0000","last_fullsized":"2024-04-16T07:42:45.325226+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.333328+0000","last_change":"2024-04-16T07:42:45.334636+0000","last_active":"2024-04-16T07:42:45.333328+0000","last_peered":"2024-04-16T07:42:45.333328+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.329643+0000","last_became_peered":"2024-04-16T07:42:45.329643+0000","last_unstale":"2024-04-16T07:42:45.333328+0000","last_undegraded":"2024-04-16T07:42:45.333328+0000","last_fullsized":"2024-04-16T07:42:45.333328+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:42:45.348344+0000","last_change":"2024-04-16T07:42:45.350220+0000","last_active":"2024-04-16T07:42:45.348344+0000","last_peered":"2024-04-16T07:42:45.348344+0000","last_clean":"2024-04-16T07:42:44.238048+0000","last_became_active":"2024-04-16T07:42:45.343569+0000","last_became_peered":"2024-04-16T07:42:45.343569+0000","last_unstale":"2024-04-16T07:42:45.348344+0000","last_undegraded":"2024-04-16T07:42:45.348344+0000","last_fullsized":"2024-04-16T07:42:45.348344+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T07:42:46.271689+0000","last_change":"2024-04-16T07:42:43.287780+0000","last_active":"2024-04-16T07:42:46.271689+0000","last_peered":"2024-04-16T07:42:46.271689+0000","last_clean":"2024-04-16T07:42:46.271689+0000","last_became_active":"2024-04-16T07:42:43.282375+0000","last_became_peered":"2024-04-16T07:42:43.282375+0000","last_unstale":"2024-04-16T07:42:46.271689+0000","last_undegraded":"2024-04-16T07:42:46.271689+0000","last_fullsized":"2024-04-16T07:42:46.271689+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_clean_scrub_stamp":"2024-04-16T07:42:42.229959+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:42:56.294 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_0' created 2024-04-16T07:42:56.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-16T07:42:56.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:56.457+0000 7f6fde99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:56.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:56.471+0000 7f6fde99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:57.300 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-16T07:42:57.703 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:42:57.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:57.814+0000 7f59564ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:57.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:42:57.820+0000 7f59564ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:42:58.324 DEBUG:teuthology.orchestra.run.smithi031:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 150 write' 2024-04-16T07:42:58.327 INFO:teuthology.run_tasks:Running task admin_socket... 2024-04-16T07:42:58.340 DEBUG:tasks.admin_socket:Running admin socket tests on client.0 2024-04-16T07:42:58.341 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -- /home/ubuntu/cephtest/admin_socket_client.0 && while test '!' -e /var/run/ceph/ceph-client.0.asok ; do sleep 1 ; done 2024-04-16T07:42:58.391 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:hints = 1 2024-04-16T07:42:58.391 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 150 seconds or 0 objects 2024-04-16T07:42:58.391 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Object prefix: benchmark_data_smithi031_32265 2024-04-16T07:42:58.392 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:42:58.392 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 0 0 0 0 0 0 - 0 2024-04-16T07:42:59.358 DEBUG:tasks.admin_socket:Testing objecter_requests with config {'test': 'http://git.ceph.com/?p={repo};a=blob_plain;f=src/test/admin_socket/objecter_requests;hb={branch}', 'branch': 'wip-matanb-crimson-replicated-april-15'} 2024-04-16T07:42:59.358 DEBUG:teuthology.orchestra.run.smithi031:> wget -q -O /home/ubuntu/cephtest/admin_socket_client.0/objecter_requests -- 'http://git.ceph.com/?p=ceph-ci.git;a=blob_plain;f=src/test/admin_socket/objecter_requests;hb=wip-matanb-crimson-replicated-april-15' && chmod u=rx -- /home/ubuntu/cephtest/admin_socket_client.0/objecter_requests 2024-04-16T07:42:59.392 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 1 16 209 193 12.0617 12.0625 0.0872023 0.0791429 2024-04-16T07:42:59.762 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-client.0.asok objecter_requests 2024-04-16T07:42:59.927 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:42:59.951 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-16T07:42:59.951 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-04-16T07:42:59.951 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:42:59.951 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 319, 2024-04-16T07:42:59.952 INFO:teuthology.orchestra.run.smithi031.stdout: "pg": "3.b7ca0c7a", 2024-04-16T07:42:59.952 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-04-16T07:42:59.952 INFO:teuthology.orchestra.run.smithi031.stdout: "object_id": "benchmark_data_smithi031_32265_object318", 2024-04-16T07:42:59.952 INFO:teuthology.orchestra.run.smithi031.stdout: "object_locator": "@3", 2024-04-16T07:42:59.952 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_id": "benchmark_data_smithi031_32265_object318", 2024-04-16T07:42:59.952 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_locator": "@3", 2024-04-16T07:42:59.952 INFO:teuthology.orchestra.run.smithi031.stdout: "paused": 0, 2024-04-16T07:42:59.953 INFO:teuthology.orchestra.run.smithi031.stdout: "used_replica": 0, 2024-04-16T07:42:59.953 INFO:teuthology.orchestra.run.smithi031.stdout: "precalc_pgid": 0, 2024-04-16T07:42:59.953 INFO:teuthology.orchestra.run.smithi031.stdout: "last_sent": "696.233581s", 2024-04-16T07:42:59.953 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.029999518999999999, 2024-04-16T07:42:59.953 INFO:teuthology.orchestra.run.smithi031.stdout: "attempts": 1, 2024-04-16T07:42:59.953 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": "head", 2024-04-16T07:42:59.954 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_context": "0=[]", 2024-04-16T07:42:59.954 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-04-16T07:42:59.921954+0000", 2024-04-16T07:42:59.954 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ops": [ 2024-04-16T07:42:59.954 INFO:teuthology.orchestra.run.smithi031.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-16T07:42:59.954 INFO:teuthology.orchestra.run.smithi031.stdout: "write 0~65536 in=65536b" 2024-04-16T07:42:59.954 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-16T07:42:59.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:42:59.955 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:42:59.955 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 324, 2024-04-16T07:42:59.955 INFO:teuthology.orchestra.run.smithi031.stdout: "pg": "3.4a5a4b7a", 2024-04-16T07:42:59.955 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-04-16T07:42:59.955 INFO:teuthology.orchestra.run.smithi031.stdout: "object_id": "benchmark_data_smithi031_32265_object323", 2024-04-16T07:42:59.955 INFO:teuthology.orchestra.run.smithi031.stdout: "object_locator": "@3", 2024-04-16T07:42:59.956 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_id": "benchmark_data_smithi031_32265_object323", 2024-04-16T07:42:59.956 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_locator": "@3", 2024-04-16T07:42:59.956 INFO:teuthology.orchestra.run.smithi031.stdout: "paused": 0, 2024-04-16T07:42:59.956 INFO:teuthology.orchestra.run.smithi031.stdout: "used_replica": 0, 2024-04-16T07:42:59.956 INFO:teuthology.orchestra.run.smithi031.stdout: "precalc_pgid": 0, 2024-04-16T07:42:59.956 INFO:teuthology.orchestra.run.smithi031.stdout: "last_sent": "696.248580s", 2024-04-16T07:42:59.956 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.014999759999999999, 2024-04-16T07:42:59.957 INFO:teuthology.orchestra.run.smithi031.stdout: "attempts": 1, 2024-04-16T07:42:59.957 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": "head", 2024-04-16T07:42:59.957 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_context": "0=[]", 2024-04-16T07:42:59.957 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-04-16T07:42:59.937331+0000", 2024-04-16T07:42:59.957 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ops": [ 2024-04-16T07:42:59.957 INFO:teuthology.orchestra.run.smithi031.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-16T07:42:59.957 INFO:teuthology.orchestra.run.smithi031.stdout: "write 0~65536 in=65536b" 2024-04-16T07:42:59.958 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-16T07:42:59.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:42:59.958 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:42:59.958 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 326, 2024-04-16T07:42:59.958 INFO:teuthology.orchestra.run.smithi031.stdout: "pg": "3.331876dc", 2024-04-16T07:42:59.958 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-04-16T07:42:59.959 INFO:teuthology.orchestra.run.smithi031.stdout: "object_id": "benchmark_data_smithi031_32265_object325", 2024-04-16T07:42:59.959 INFO:teuthology.orchestra.run.smithi031.stdout: "object_locator": "@3", 2024-04-16T07:42:59.959 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_id": "benchmark_data_smithi031_32265_object325", 2024-04-16T07:42:59.959 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_locator": "@3", 2024-04-16T07:42:59.959 INFO:teuthology.orchestra.run.smithi031.stdout: "paused": 0, 2024-04-16T07:42:59.959 INFO:teuthology.orchestra.run.smithi031.stdout: "used_replica": 0, 2024-04-16T07:42:59.959 INFO:teuthology.orchestra.run.smithi031.stdout: "precalc_pgid": 0, 2024-04-16T07:42:59.960 INFO:teuthology.orchestra.run.smithi031.stdout: "last_sent": "696.248580s", 2024-04-16T07:42:59.960 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.014999759999999999, 2024-04-16T07:42:59.960 INFO:teuthology.orchestra.run.smithi031.stdout: "attempts": 1, 2024-04-16T07:42:59.960 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": "head", 2024-04-16T07:42:59.960 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_context": "0=[]", 2024-04-16T07:42:59.960 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-04-16T07:42:59.937546+0000", 2024-04-16T07:42:59.960 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ops": [ 2024-04-16T07:42:59.961 INFO:teuthology.orchestra.run.smithi031.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-16T07:42:59.961 INFO:teuthology.orchestra.run.smithi031.stdout: "write 0~65536 in=65536b" 2024-04-16T07:42:59.961 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-16T07:42:59.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:42:59.961 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:42:59.961 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 327, 2024-04-16T07:42:59.961 INFO:teuthology.orchestra.run.smithi031.stdout: "pg": "3.382f22af", 2024-04-16T07:42:59.962 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-04-16T07:42:59.962 INFO:teuthology.orchestra.run.smithi031.stdout: "object_id": "benchmark_data_smithi031_32265_object326", 2024-04-16T07:42:59.962 INFO:teuthology.orchestra.run.smithi031.stdout: "object_locator": "@3", 2024-04-16T07:42:59.962 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_id": "benchmark_data_smithi031_32265_object326", 2024-04-16T07:42:59.962 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_locator": "@3", 2024-04-16T07:42:59.962 INFO:teuthology.orchestra.run.smithi031.stdout: "paused": 0, 2024-04-16T07:42:59.963 INFO:teuthology.orchestra.run.smithi031.stdout: "used_replica": 0, 2024-04-16T07:42:59.963 INFO:teuthology.orchestra.run.smithi031.stdout: "precalc_pgid": 0, 2024-04-16T07:42:59.963 INFO:teuthology.orchestra.run.smithi031.stdout: "last_sent": "696.250580s", 2024-04-16T07:42:59.963 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.012999792, 2024-04-16T07:42:59.963 INFO:teuthology.orchestra.run.smithi031.stdout: "attempts": 1, 2024-04-16T07:42:59.963 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": "head", 2024-04-16T07:42:59.963 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_context": "0=[]", 2024-04-16T07:42:59.964 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-04-16T07:42:59.939691+0000", 2024-04-16T07:42:59.964 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ops": [ 2024-04-16T07:42:59.964 INFO:teuthology.orchestra.run.smithi031.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-16T07:42:59.964 INFO:teuthology.orchestra.run.smithi031.stdout: "write 0~65536 in=65536b" 2024-04-16T07:42:59.964 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-16T07:42:59.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:42:59.964 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:42:59.965 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 308, 2024-04-16T07:42:59.965 INFO:teuthology.orchestra.run.smithi031.stdout: "pg": "3.2a0a408", 2024-04-16T07:42:59.965 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-04-16T07:42:59.965 INFO:teuthology.orchestra.run.smithi031.stdout: "object_id": "benchmark_data_smithi031_32265_object307", 2024-04-16T07:42:59.965 INFO:teuthology.orchestra.run.smithi031.stdout: "object_locator": "@3", 2024-04-16T07:42:59.965 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_id": "benchmark_data_smithi031_32265_object307", 2024-04-16T07:42:59.966 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_locator": "@3", 2024-04-16T07:42:59.966 INFO:teuthology.orchestra.run.smithi031.stdout: "paused": 0, 2024-04-16T07:42:59.966 INFO:teuthology.orchestra.run.smithi031.stdout: "used_replica": 0, 2024-04-16T07:42:59.966 INFO:teuthology.orchestra.run.smithi031.stdout: "precalc_pgid": 0, 2024-04-16T07:42:59.966 INFO:teuthology.orchestra.run.smithi031.stdout: "last_sent": "696.166582s", 2024-04-16T07:42:59.966 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.096998445000000003, 2024-04-16T07:42:59.967 INFO:teuthology.orchestra.run.smithi031.stdout: "attempts": 1, 2024-04-16T07:42:59.967 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": "head", 2024-04-16T07:42:59.967 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_context": "0=[]", 2024-04-16T07:42:59.967 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-04-16T07:42:59.854978+0000", 2024-04-16T07:42:59.967 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ops": [ 2024-04-16T07:42:59.967 INFO:teuthology.orchestra.run.smithi031.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-16T07:42:59.967 INFO:teuthology.orchestra.run.smithi031.stdout: "write 0~65536 in=65536b" 2024-04-16T07:42:59.968 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-16T07:42:59.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:42:59.968 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:42:59.968 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 316, 2024-04-16T07:42:59.968 INFO:teuthology.orchestra.run.smithi031.stdout: "pg": "3.6f6f4b65", 2024-04-16T07:42:59.968 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-04-16T07:42:59.968 INFO:teuthology.orchestra.run.smithi031.stdout: "object_id": "benchmark_data_smithi031_32265_object315", 2024-04-16T07:42:59.969 INFO:teuthology.orchestra.run.smithi031.stdout: "object_locator": "@3", 2024-04-16T07:42:59.969 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_id": "benchmark_data_smithi031_32265_object315", 2024-04-16T07:42:59.969 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_locator": "@3", 2024-04-16T07:42:59.969 INFO:teuthology.orchestra.run.smithi031.stdout: "paused": 0, 2024-04-16T07:42:59.969 INFO:teuthology.orchestra.run.smithi031.stdout: "used_replica": 0, 2024-04-16T07:42:59.969 INFO:teuthology.orchestra.run.smithi031.stdout: "precalc_pgid": 0, 2024-04-16T07:42:59.969 INFO:teuthology.orchestra.run.smithi031.stdout: "last_sent": "696.207581s", 2024-04-16T07:42:59.970 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.055999102000000002, 2024-04-16T07:42:59.970 INFO:teuthology.orchestra.run.smithi031.stdout: "attempts": 1, 2024-04-16T07:42:59.970 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": "head", 2024-04-16T07:42:59.970 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_context": "0=[]", 2024-04-16T07:42:59.970 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-04-16T07:42:59.895962+0000", 2024-04-16T07:42:59.970 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ops": [ 2024-04-16T07:42:59.971 INFO:teuthology.orchestra.run.smithi031.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-16T07:42:59.971 INFO:teuthology.orchestra.run.smithi031.stdout: "write 0~65536 in=65536b" 2024-04-16T07:42:59.971 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-16T07:42:59.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:42:59.971 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:42:59.971 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 323, 2024-04-16T07:42:59.971 INFO:teuthology.orchestra.run.smithi031.stdout: "pg": "3.1b34c518", 2024-04-16T07:42:59.972 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-04-16T07:42:59.972 INFO:teuthology.orchestra.run.smithi031.stdout: "object_id": "benchmark_data_smithi031_32265_object322", 2024-04-16T07:42:59.972 INFO:teuthology.orchestra.run.smithi031.stdout: "object_locator": "@3", 2024-04-16T07:42:59.972 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_id": "benchmark_data_smithi031_32265_object322", 2024-04-16T07:42:59.972 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_locator": "@3", 2024-04-16T07:42:59.972 INFO:teuthology.orchestra.run.smithi031.stdout: "paused": 0, 2024-04-16T07:42:59.972 INFO:teuthology.orchestra.run.smithi031.stdout: "used_replica": 0, 2024-04-16T07:42:59.973 INFO:teuthology.orchestra.run.smithi031.stdout: "precalc_pgid": 0, 2024-04-16T07:42:59.973 INFO:teuthology.orchestra.run.smithi031.stdout: "last_sent": "696.242580s", 2024-04-16T07:42:59.973 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.020999662999999998, 2024-04-16T07:42:59.973 INFO:teuthology.orchestra.run.smithi031.stdout: "attempts": 1, 2024-04-16T07:42:59.973 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": "head", 2024-04-16T07:42:59.973 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_context": "0=[]", 2024-04-16T07:42:59.974 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-04-16T07:42:59.930924+0000", 2024-04-16T07:42:59.974 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ops": [ 2024-04-16T07:42:59.974 INFO:teuthology.orchestra.run.smithi031.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-16T07:42:59.974 INFO:teuthology.orchestra.run.smithi031.stdout: "write 0~65536 in=65536b" 2024-04-16T07:42:59.974 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-16T07:42:59.974 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:42:59.974 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:42:59.975 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 317, 2024-04-16T07:42:59.975 INFO:teuthology.orchestra.run.smithi031.stdout: "pg": "3.b2e58df4", 2024-04-16T07:42:59.975 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-04-16T07:42:59.975 INFO:teuthology.orchestra.run.smithi031.stdout: "object_id": "benchmark_data_smithi031_32265_object316", 2024-04-16T07:42:59.975 INFO:teuthology.orchestra.run.smithi031.stdout: "object_locator": "@3", 2024-04-16T07:42:59.975 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_id": "benchmark_data_smithi031_32265_object316", 2024-04-16T07:42:59.975 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_locator": "@3", 2024-04-16T07:42:59.976 INFO:teuthology.orchestra.run.smithi031.stdout: "paused": 0, 2024-04-16T07:42:59.976 INFO:teuthology.orchestra.run.smithi031.stdout: "used_replica": 0, 2024-04-16T07:42:59.976 INFO:teuthology.orchestra.run.smithi031.stdout: "precalc_pgid": 0, 2024-04-16T07:42:59.976 INFO:teuthology.orchestra.run.smithi031.stdout: "last_sent": "696.231581s", 2024-04-16T07:42:59.976 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.031999487, 2024-04-16T07:42:59.976 INFO:teuthology.orchestra.run.smithi031.stdout: "attempts": 1, 2024-04-16T07:42:59.976 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": "head", 2024-04-16T07:42:59.977 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_context": "0=[]", 2024-04-16T07:42:59.977 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-04-16T07:42:59.920278+0000", 2024-04-16T07:42:59.977 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ops": [ 2024-04-16T07:42:59.977 INFO:teuthology.orchestra.run.smithi031.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-16T07:42:59.977 INFO:teuthology.orchestra.run.smithi031.stdout: "write 0~65536 in=65536b" 2024-04-16T07:42:59.977 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-16T07:42:59.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:42:59.978 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:42:59.978 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 318, 2024-04-16T07:42:59.978 INFO:teuthology.orchestra.run.smithi031.stdout: "pg": "3.80950ffe", 2024-04-16T07:42:59.978 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-04-16T07:42:59.978 INFO:teuthology.orchestra.run.smithi031.stdout: "object_id": "benchmark_data_smithi031_32265_object317", 2024-04-16T07:42:59.978 INFO:teuthology.orchestra.run.smithi031.stdout: "object_locator": "@3", 2024-04-16T07:42:59.978 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_id": "benchmark_data_smithi031_32265_object317", 2024-04-16T07:42:59.979 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_locator": "@3", 2024-04-16T07:42:59.979 INFO:teuthology.orchestra.run.smithi031.stdout: "paused": 0, 2024-04-16T07:42:59.979 INFO:teuthology.orchestra.run.smithi031.stdout: "used_replica": 0, 2024-04-16T07:42:59.979 INFO:teuthology.orchestra.run.smithi031.stdout: "precalc_pgid": 0, 2024-04-16T07:42:59.979 INFO:teuthology.orchestra.run.smithi031.stdout: "last_sent": "696.232581s", 2024-04-16T07:42:59.979 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.030999503000000001, 2024-04-16T07:42:59.979 INFO:teuthology.orchestra.run.smithi031.stdout: "attempts": 1, 2024-04-16T07:42:59.980 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": "head", 2024-04-16T07:42:59.980 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_context": "0=[]", 2024-04-16T07:42:59.980 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-04-16T07:42:59.921708+0000", 2024-04-16T07:42:59.980 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ops": [ 2024-04-16T07:42:59.980 INFO:teuthology.orchestra.run.smithi031.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-16T07:42:59.980 INFO:teuthology.orchestra.run.smithi031.stdout: "write 0~65536 in=65536b" 2024-04-16T07:42:59.980 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-16T07:42:59.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:42:59.981 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:42:59.981 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 320, 2024-04-16T07:42:59.981 INFO:teuthology.orchestra.run.smithi031.stdout: "pg": "3.12839524", 2024-04-16T07:42:59.981 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-04-16T07:42:59.981 INFO:teuthology.orchestra.run.smithi031.stdout: "object_id": "benchmark_data_smithi031_32265_object319", 2024-04-16T07:42:59.981 INFO:teuthology.orchestra.run.smithi031.stdout: "object_locator": "@3", 2024-04-16T07:42:59.982 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_id": "benchmark_data_smithi031_32265_object319", 2024-04-16T07:42:59.982 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_locator": "@3", 2024-04-16T07:42:59.982 INFO:teuthology.orchestra.run.smithi031.stdout: "paused": 0, 2024-04-16T07:42:59.982 INFO:teuthology.orchestra.run.smithi031.stdout: "used_replica": 0, 2024-04-16T07:42:59.982 INFO:teuthology.orchestra.run.smithi031.stdout: "precalc_pgid": 0, 2024-04-16T07:42:59.982 INFO:teuthology.orchestra.run.smithi031.stdout: "last_sent": "696.233581s", 2024-04-16T07:42:59.983 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.029999518999999999, 2024-04-16T07:42:59.983 INFO:teuthology.orchestra.run.smithi031.stdout: "attempts": 1, 2024-04-16T07:42:59.983 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": "head", 2024-04-16T07:42:59.983 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_context": "0=[]", 2024-04-16T07:42:59.983 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-04-16T07:42:59.921977+0000", 2024-04-16T07:42:59.983 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ops": [ 2024-04-16T07:42:59.983 INFO:teuthology.orchestra.run.smithi031.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-16T07:42:59.983 INFO:teuthology.orchestra.run.smithi031.stdout: "write 0~65536 in=65536b" 2024-04-16T07:42:59.984 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-16T07:42:59.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:42:59.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:42:59.984 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 321, 2024-04-16T07:42:59.984 INFO:teuthology.orchestra.run.smithi031.stdout: "pg": "3.ef0c5bc4", 2024-04-16T07:42:59.984 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-04-16T07:42:59.984 INFO:teuthology.orchestra.run.smithi031.stdout: "object_id": "benchmark_data_smithi031_32265_object320", 2024-04-16T07:42:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: "object_locator": "@3", 2024-04-16T07:42:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_id": "benchmark_data_smithi031_32265_object320", 2024-04-16T07:42:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_locator": "@3", 2024-04-16T07:42:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: "paused": 0, 2024-04-16T07:42:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: "used_replica": 0, 2024-04-16T07:42:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: "precalc_pgid": 0, 2024-04-16T07:42:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: "last_sent": "696.233581s", 2024-04-16T07:42:59.986 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.029999518999999999, 2024-04-16T07:42:59.986 INFO:teuthology.orchestra.run.smithi031.stdout: "attempts": 1, 2024-04-16T07:42:59.986 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": "head", 2024-04-16T07:42:59.986 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_context": "0=[]", 2024-04-16T07:42:59.986 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-04-16T07:42:59.921992+0000", 2024-04-16T07:42:59.986 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ops": [ 2024-04-16T07:42:59.987 INFO:teuthology.orchestra.run.smithi031.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-16T07:42:59.987 INFO:teuthology.orchestra.run.smithi031.stdout: "write 0~65536 in=65536b" 2024-04-16T07:42:59.987 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-16T07:42:59.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:42:59.987 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:42:59.987 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 322, 2024-04-16T07:42:59.987 INFO:teuthology.orchestra.run.smithi031.stdout: "pg": "3.ba7e00b4", 2024-04-16T07:42:59.987 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-04-16T07:42:59.988 INFO:teuthology.orchestra.run.smithi031.stdout: "object_id": "benchmark_data_smithi031_32265_object321", 2024-04-16T07:42:59.988 INFO:teuthology.orchestra.run.smithi031.stdout: "object_locator": "@3", 2024-04-16T07:42:59.988 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_id": "benchmark_data_smithi031_32265_object321", 2024-04-16T07:42:59.988 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_locator": "@3", 2024-04-16T07:42:59.988 INFO:teuthology.orchestra.run.smithi031.stdout: "paused": 0, 2024-04-16T07:42:59.988 INFO:teuthology.orchestra.run.smithi031.stdout: "used_replica": 0, 2024-04-16T07:42:59.988 INFO:teuthology.orchestra.run.smithi031.stdout: "precalc_pgid": 0, 2024-04-16T07:42:59.989 INFO:teuthology.orchestra.run.smithi031.stdout: "last_sent": "696.235581s", 2024-04-16T07:42:59.989 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.027999551000000001, 2024-04-16T07:42:59.989 INFO:teuthology.orchestra.run.smithi031.stdout: "attempts": 1, 2024-04-16T07:42:59.989 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": "head", 2024-04-16T07:42:59.989 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_context": "0=[]", 2024-04-16T07:42:59.989 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-04-16T07:42:59.924426+0000", 2024-04-16T07:42:59.989 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ops": [ 2024-04-16T07:42:59.990 INFO:teuthology.orchestra.run.smithi031.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-16T07:42:59.990 INFO:teuthology.orchestra.run.smithi031.stdout: "write 0~65536 in=65536b" 2024-04-16T07:42:59.990 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-16T07:42:59.990 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:42:59.990 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:42:59.990 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 325, 2024-04-16T07:42:59.991 INFO:teuthology.orchestra.run.smithi031.stdout: "pg": "3.852091be", 2024-04-16T07:42:59.991 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-04-16T07:42:59.991 INFO:teuthology.orchestra.run.smithi031.stdout: "object_id": "benchmark_data_smithi031_32265_object324", 2024-04-16T07:42:59.991 INFO:teuthology.orchestra.run.smithi031.stdout: "object_locator": "@3", 2024-04-16T07:42:59.991 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_id": "benchmark_data_smithi031_32265_object324", 2024-04-16T07:42:59.991 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_locator": "@3", 2024-04-16T07:42:59.991 INFO:teuthology.orchestra.run.smithi031.stdout: "paused": 0, 2024-04-16T07:42:59.992 INFO:teuthology.orchestra.run.smithi031.stdout: "used_replica": 0, 2024-04-16T07:42:59.992 INFO:teuthology.orchestra.run.smithi031.stdout: "precalc_pgid": 0, 2024-04-16T07:42:59.992 INFO:teuthology.orchestra.run.smithi031.stdout: "last_sent": "696.248580s", 2024-04-16T07:42:59.992 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.014999759999999999, 2024-04-16T07:42:59.992 INFO:teuthology.orchestra.run.smithi031.stdout: "attempts": 1, 2024-04-16T07:42:59.992 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": "head", 2024-04-16T07:42:59.992 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_context": "0=[]", 2024-04-16T07:42:59.993 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-04-16T07:42:59.937436+0000", 2024-04-16T07:42:59.993 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ops": [ 2024-04-16T07:42:59.993 INFO:teuthology.orchestra.run.smithi031.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-16T07:42:59.993 INFO:teuthology.orchestra.run.smithi031.stdout: "write 0~65536 in=65536b" 2024-04-16T07:42:59.993 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-16T07:42:59.993 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:42:59.993 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:42:59.994 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 330, 2024-04-16T07:42:59.994 INFO:teuthology.orchestra.run.smithi031.stdout: "pg": "3.3b3cbcb4", 2024-04-16T07:42:59.994 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-04-16T07:42:59.994 INFO:teuthology.orchestra.run.smithi031.stdout: "object_id": "benchmark_data_smithi031_32265_object329", 2024-04-16T07:42:59.994 INFO:teuthology.orchestra.run.smithi031.stdout: "object_locator": "@3", 2024-04-16T07:42:59.994 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_id": "benchmark_data_smithi031_32265_object329", 2024-04-16T07:42:59.994 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_locator": "@3", 2024-04-16T07:42:59.994 INFO:teuthology.orchestra.run.smithi031.stdout: "paused": 0, 2024-04-16T07:42:59.995 INFO:teuthology.orchestra.run.smithi031.stdout: "used_replica": 0, 2024-04-16T07:42:59.995 INFO:teuthology.orchestra.run.smithi031.stdout: "precalc_pgid": 0, 2024-04-16T07:42:59.995 INFO:teuthology.orchestra.run.smithi031.stdout: "last_sent": "696.262580s", 2024-04-16T07:42:59.995 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.00099998399999999999, 2024-04-16T07:42:59.995 INFO:teuthology.orchestra.run.smithi031.stdout: "attempts": 1, 2024-04-16T07:42:59.995 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": "head", 2024-04-16T07:42:59.995 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_context": "0=[]", 2024-04-16T07:42:59.996 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-04-16T07:42:59.950885+0000", 2024-04-16T07:42:59.996 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ops": [ 2024-04-16T07:42:59.996 INFO:teuthology.orchestra.run.smithi031.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-16T07:42:59.996 INFO:teuthology.orchestra.run.smithi031.stdout: "write 0~65536 in=65536b" 2024-04-16T07:42:59.996 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-16T07:42:59.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:42:59.996 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:42:59.997 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 328, 2024-04-16T07:42:59.997 INFO:teuthology.orchestra.run.smithi031.stdout: "pg": "3.f042968b", 2024-04-16T07:42:59.997 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-04-16T07:42:59.997 INFO:teuthology.orchestra.run.smithi031.stdout: "object_id": "benchmark_data_smithi031_32265_object327", 2024-04-16T07:42:59.997 INFO:teuthology.orchestra.run.smithi031.stdout: "object_locator": "@3", 2024-04-16T07:42:59.997 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_id": "benchmark_data_smithi031_32265_object327", 2024-04-16T07:42:59.997 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_locator": "@3", 2024-04-16T07:42:59.998 INFO:teuthology.orchestra.run.smithi031.stdout: "paused": 0, 2024-04-16T07:42:59.998 INFO:teuthology.orchestra.run.smithi031.stdout: "used_replica": 0, 2024-04-16T07:42:59.998 INFO:teuthology.orchestra.run.smithi031.stdout: "precalc_pgid": 0, 2024-04-16T07:42:59.998 INFO:teuthology.orchestra.run.smithi031.stdout: "last_sent": "696.260580s", 2024-04-16T07:42:59.998 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0029999520000000002, 2024-04-16T07:42:59.998 INFO:teuthology.orchestra.run.smithi031.stdout: "attempts": 1, 2024-04-16T07:42:59.998 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": "head", 2024-04-16T07:42:59.999 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_context": "0=[]", 2024-04-16T07:42:59.999 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-04-16T07:42:59.949580+0000", 2024-04-16T07:42:59.999 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ops": [ 2024-04-16T07:42:59.999 INFO:teuthology.orchestra.run.smithi031.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-16T07:42:59.999 INFO:teuthology.orchestra.run.smithi031.stdout: "write 0~65536 in=65536b" 2024-04-16T07:42:59.999 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-16T07:42:59.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:43:00.000 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:43:00.000 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 329, 2024-04-16T07:43:00.000 INFO:teuthology.orchestra.run.smithi031.stdout: "pg": "3.c67f1207", 2024-04-16T07:43:00.000 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-04-16T07:43:00.000 INFO:teuthology.orchestra.run.smithi031.stdout: "object_id": "benchmark_data_smithi031_32265_object328", 2024-04-16T07:43:00.000 INFO:teuthology.orchestra.run.smithi031.stdout: "object_locator": "@3", 2024-04-16T07:43:00.000 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_id": "benchmark_data_smithi031_32265_object328", 2024-04-16T07:43:00.001 INFO:teuthology.orchestra.run.smithi031.stdout: "target_object_locator": "@3", 2024-04-16T07:43:00.001 INFO:teuthology.orchestra.run.smithi031.stdout: "paused": 0, 2024-04-16T07:43:00.001 INFO:teuthology.orchestra.run.smithi031.stdout: "used_replica": 0, 2024-04-16T07:43:00.001 INFO:teuthology.orchestra.run.smithi031.stdout: "precalc_pgid": 0, 2024-04-16T07:43:00.001 INFO:teuthology.orchestra.run.smithi031.stdout: "last_sent": "696.260580s", 2024-04-16T07:43:00.001 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0029999520000000002, 2024-04-16T07:43:00.002 INFO:teuthology.orchestra.run.smithi031.stdout: "attempts": 1, 2024-04-16T07:43:00.002 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": "head", 2024-04-16T07:43:00.002 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_context": "0=[]", 2024-04-16T07:43:00.002 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-04-16T07:42:59.949622+0000", 2024-04-16T07:43:00.002 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ops": [ 2024-04-16T07:43:00.002 INFO:teuthology.orchestra.run.smithi031.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-04-16T07:43:00.002 INFO:teuthology.orchestra.run.smithi031.stdout: "write 0~65536 in=65536b" 2024-04-16T07:43:00.003 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-16T07:43:00.003 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:43:00.003 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-16T07:43:00.003 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_ops": [], 2024-04-16T07:43:00.003 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_ops": [], 2024-04-16T07:43:00.003 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_stat_ops": [], 2024-04-16T07:43:00.003 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_ops": [], 2024-04-16T07:43:00.004 INFO:teuthology.orchestra.run.smithi031.stdout: "command_ops": [] 2024-04-16T07:43:00.004 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-16T07:43:00.005 DEBUG:tasks.admin_socket:admin socket command objecter_requests returned { "ops": [ { "tid": 319, "pg": "3.b7ca0c7a", "osd": 0, "object_id": "benchmark_data_smithi031_32265_object318", "object_locator": "@3", "target_object_id": "benchmark_data_smithi031_32265_object318", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "696.233581s", "age": 0.029999518999999999, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-16T07:42:59.921954+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 324, "pg": "3.4a5a4b7a", "osd": 0, "object_id": "benchmark_data_smithi031_32265_object323", "object_locator": "@3", "target_object_id": "benchmark_data_smithi031_32265_object323", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "696.248580s", "age": 0.014999759999999999, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-16T07:42:59.937331+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 326, "pg": "3.331876dc", "osd": 0, "object_id": "benchmark_data_smithi031_32265_object325", "object_locator": "@3", "target_object_id": "benchmark_data_smithi031_32265_object325", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "696.248580s", "age": 0.014999759999999999, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-16T07:42:59.937546+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 327, "pg": "3.382f22af", "osd": 0, "object_id": "benchmark_data_smithi031_32265_object326", "object_locator": "@3", "target_object_id": "benchmark_data_smithi031_32265_object326", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "696.250580s", "age": 0.012999792, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-16T07:42:59.939691+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 308, "pg": "3.2a0a408", "osd": 1, "object_id": "benchmark_data_smithi031_32265_object307", "object_locator": "@3", "target_object_id": "benchmark_data_smithi031_32265_object307", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "696.166582s", "age": 0.096998445000000003, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-16T07:42:59.854978+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 316, "pg": "3.6f6f4b65", "osd": 1, "object_id": "benchmark_data_smithi031_32265_object315", "object_locator": "@3", "target_object_id": "benchmark_data_smithi031_32265_object315", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "696.207581s", "age": 0.055999102000000002, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-16T07:42:59.895962+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 323, "pg": "3.1b34c518", "osd": 1, "object_id": "benchmark_data_smithi031_32265_object322", "object_locator": "@3", "target_object_id": "benchmark_data_smithi031_32265_object322", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "696.242580s", "age": 0.020999662999999998, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-16T07:42:59.930924+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 317, "pg": "3.b2e58df4", "osd": 2, "object_id": "benchmark_data_smithi031_32265_object316", "object_locator": "@3", "target_object_id": "benchmark_data_smithi031_32265_object316", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "696.231581s", "age": 0.031999487, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-16T07:42:59.920278+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 318, "pg": "3.80950ffe", "osd": 2, "object_id": "benchmark_data_smithi031_32265_object317", "object_locator": "@3", "target_object_id": "benchmark_data_smithi031_32265_object317", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "696.232581s", "age": 0.030999503000000001, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-16T07:42:59.921708+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 320, "pg": "3.12839524", "osd": 2, "object_id": "benchmark_data_smithi031_32265_object319", "object_locator": "@3", "target_object_id": "benchmark_data_smithi031_32265_object319", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "696.233581s", "age": 0.029999518999999999, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-16T07:42:59.921977+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 321, "pg": "3.ef0c5bc4", "osd": 2, "object_id": "benchmark_data_smithi031_32265_object320", "object_locator": "@3", "target_object_id": "benchmark_data_smithi031_32265_object320", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "696.233581s", "age": 0.029999518999999999, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-16T07:42:59.921992+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 322, "pg": "3.ba7e00b4", "osd": 2, "object_id": "benchmark_data_smithi031_32265_object321", "object_locator": "@3", "target_object_id": "benchmark_data_smithi031_32265_object321", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "696.235581s", "age": 0.027999551000000001, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-16T07:42:59.924426+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 325, "pg": "3.852091be", "osd": 2, "object_id": "benchmark_data_smithi031_32265_object324", "object_locator": "@3", "target_object_id": "benchmark_data_smithi031_32265_object324", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "696.248580s", "age": 0.014999759999999999, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-16T07:42:59.937436+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 330, "pg": "3.3b3cbcb4", "osd": 2, "object_id": "benchmark_data_smithi031_32265_object329", "object_locator": "@3", "target_object_id": "benchmark_data_smithi031_32265_object329", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "696.262580s", "age": 0.00099998399999999999, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-16T07:42:59.950885+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 328, "pg": "3.f042968b", "osd": 3, "object_id": "benchmark_data_smithi031_32265_object327", "object_locator": "@3", "target_object_id": "benchmark_data_smithi031_32265_object327", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "696.260580s", "age": 0.0029999520000000002, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-16T07:42:59.949580+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 329, "pg": "3.c67f1207", "osd": 3, "object_id": "benchmark_data_smithi031_32265_object328", "object_locator": "@3", "target_object_id": "benchmark_data_smithi031_32265_object328", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "696.260580s", "age": 0.0029999520000000002, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-04-16T07:42:59.949622+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] } ], "linger_ops": [], "pool_ops": [], "pool_stat_ops": [], "statfs_ops": [], "command_ops": [] } 2024-04-16T07:43:00.006 DEBUG:teuthology.orchestra.run.smithi031:> /home/ubuntu/cephtest/admin_socket_client.0/objecter_requests 2024-04-16T07:43:00.096 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/admin_socket_client.0 2024-04-16T07:43:00.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:00.097+0000 7fde5caed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:00.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:00.109+0000 7fde5caed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:00.123 DEBUG:teuthology.parallel:result is None 2024-04-16T07:43:00.124 DEBUG:teuthology.run_tasks:Unwinding manager radosbench 2024-04-16T07:43:00.138 INFO:tasks.radosbench:joining radosbench (timing out after 4800s) 2024-04-16T07:43:00.138 INFO:teuthology.orchestra.run:waiting for 4800 2024-04-16T07:43:00.392 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 2 16 424 408 12.7489 13.4375 0.0993981 0.0772587 2024-04-16T07:43:00.813 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-16T07:43:00.813 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:43:00.814 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:43:00.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:00.964+0000 7f04e166d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:00.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:00.977+0000 7f04e166d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:01.267 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:43:01.267 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-16T07:43:01.292 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-04-16T07:43:01.219735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25952800,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":524,"num_write_kb":26662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":468,"ondisk_log_size":468,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":110092,"kb_used_data":2796,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377377268,"statfs":{"total":386547056640,"available":386434322432,"internally_reserved":0,"allocated":2863104,"data_stored":35621492,"data_compressed":1688580,"data_compressed_allocated":2228224,"data_compressed_original":35405824,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25362432,"num_objects":387,"num_object_clones":0,"num_object_copies":1161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":387,"num_write_kb":24768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001090"},"pg_stats":[{"pgid":"3.f","version":"17'23","reported_seq":36,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:59.977304+0000","last_change":"2024-04-16T07:42:57.456939+0000","last_active":"2024-04-16T07:42:59.977304+0000","last_peered":"2024-04-16T07:42:59.977304+0000","last_clean":"2024-04-16T07:42:59.977304+0000","last_became_active":"2024-04-16T07:42:57.450056+0000","last_became_peered":"2024-04-16T07:42:57.450056+0000","last_unstale":"2024-04-16T07:42:59.977304+0000","last_undegraded":"2024-04-16T07:42:59.977304+0000","last_fullsized":"2024-04-16T07:42:59.977304+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"17'24","reported_seq":37,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:59.951608+0000","last_change":"2024-04-16T07:42:57.453661+0000","last_active":"2024-04-16T07:42:59.951608+0000","last_peered":"2024-04-16T07:42:59.951608+0000","last_clean":"2024-04-16T07:42:59.951608+0000","last_became_active":"2024-04-16T07:42:57.447818+0000","last_became_peered":"2024-04-16T07:42:57.447818+0000","last_unstale":"2024-04-16T07:42:59.951608+0000","last_undegraded":"2024-04-16T07:42:59.951608+0000","last_fullsized":"2024-04-16T07:42:59.951608+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'26","reported_seq":39,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:59.868565+0000","last_change":"2024-04-16T07:42:57.507834+0000","last_active":"2024-04-16T07:42:59.868565+0000","last_peered":"2024-04-16T07:42:59.868565+0000","last_clean":"2024-04-16T07:42:59.868565+0000","last_became_active":"2024-04-16T07:42:57.499895+0000","last_became_peered":"2024-04-16T07:42:57.499895+0000","last_unstale":"2024-04-16T07:42:59.868565+0000","last_undegraded":"2024-04-16T07:42:59.868565+0000","last_fullsized":"2024-04-16T07:42:59.868565+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'28","reported_seq":41,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.220396+0000","last_change":"2024-04-16T07:42:57.465377+0000","last_active":"2024-04-16T07:43:00.220396+0000","last_peered":"2024-04-16T07:43:00.220396+0000","last_clean":"2024-04-16T07:43:00.220396+0000","last_became_active":"2024-04-16T07:42:57.451654+0000","last_became_peered":"2024-04-16T07:42:57.451654+0000","last_unstale":"2024-04-16T07:43:00.220396+0000","last_undegraded":"2024-04-16T07:43:00.220396+0000","last_fullsized":"2024-04-16T07:43:00.220396+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'26","reported_seq":39,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.019395+0000","last_change":"2024-04-16T07:42:57.459819+0000","last_active":"2024-04-16T07:43:00.019395+0000","last_peered":"2024-04-16T07:43:00.019395+0000","last_clean":"2024-04-16T07:43:00.019395+0000","last_became_active":"2024-04-16T07:42:57.453218+0000","last_became_peered":"2024-04-16T07:42:57.453218+0000","last_unstale":"2024-04-16T07:43:00.019395+0000","last_undegraded":"2024-04-16T07:43:00.019395+0000","last_fullsized":"2024-04-16T07:43:00.019395+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'21","reported_seq":34,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.120346+0000","last_change":"2024-04-16T07:42:57.489933+0000","last_active":"2024-04-16T07:43:00.120346+0000","last_peered":"2024-04-16T07:43:00.120346+0000","last_clean":"2024-04-16T07:43:00.120346+0000","last_became_active":"2024-04-16T07:42:57.482517+0000","last_became_peered":"2024-04-16T07:42:57.482517+0000","last_unstale":"2024-04-16T07:43:00.120346+0000","last_undegraded":"2024-04-16T07:43:00.120346+0000","last_fullsized":"2024-04-16T07:43:00.120346+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'22","reported_seq":35,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.172645+0000","last_change":"2024-04-16T07:42:57.469015+0000","last_active":"2024-04-16T07:43:00.172645+0000","last_peered":"2024-04-16T07:43:00.172645+0000","last_clean":"2024-04-16T07:43:00.172645+0000","last_became_active":"2024-04-16T07:42:57.460537+0000","last_became_peered":"2024-04-16T07:42:57.460537+0000","last_unstale":"2024-04-16T07:43:00.172645+0000","last_undegraded":"2024-04-16T07:43:00.172645+0000","last_fullsized":"2024-04-16T07:43:00.172645+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1507328,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"17'21","reported_seq":34,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.235372+0000","last_change":"2024-04-16T07:42:57.459433+0000","last_active":"2024-04-16T07:43:00.235372+0000","last_peered":"2024-04-16T07:43:00.235372+0000","last_clean":"2024-04-16T07:43:00.235372+0000","last_became_active":"2024-04-16T07:42:57.452612+0000","last_became_peered":"2024-04-16T07:42:57.452612+0000","last_unstale":"2024-04-16T07:43:00.235372+0000","last_undegraded":"2024-04-16T07:43:00.235372+0000","last_fullsized":"2024-04-16T07:43:00.235372+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":154,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.415419+0000","last_change":"2024-04-16T07:42:43.287780+0000","last_active":"2024-04-16T07:42:58.415419+0000","last_peered":"2024-04-16T07:42:58.415419+0000","last_clean":"2024-04-16T07:42:58.415419+0000","last_became_active":"2024-04-16T07:42:43.282375+0000","last_became_peered":"2024-04-16T07:42:43.282375+0000","last_unstale":"2024-04-16T07:42:58.415419+0000","last_undegraded":"2024-04-16T07:42:58.415419+0000","last_fullsized":"2024-04-16T07:42:58.415419+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_clean_scrub_stamp":"2024-04-16T07:42:42.229959+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.331433+0000","last_change":"2024-04-16T07:42:45.350220+0000","last_active":"2024-04-16T07:42:58.331433+0000","last_peered":"2024-04-16T07:42:58.331433+0000","last_clean":"2024-04-16T07:42:58.331433+0000","last_became_active":"2024-04-16T07:42:45.343569+0000","last_became_peered":"2024-04-16T07:42:45.343569+0000","last_unstale":"2024-04-16T07:42:58.331433+0000","last_undegraded":"2024-04-16T07:42:58.331433+0000","last_fullsized":"2024-04-16T07:42:58.331433+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"17'22","reported_seq":35,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.103138+0000","last_change":"2024-04-16T07:42:57.476764+0000","last_active":"2024-04-16T07:43:00.103138+0000","last_peered":"2024-04-16T07:43:00.103138+0000","last_clean":"2024-04-16T07:43:00.103138+0000","last_became_active":"2024-04-16T07:42:57.471552+0000","last_became_peered":"2024-04-16T07:42:57.471552+0000","last_unstale":"2024-04-16T07:43:00.103138+0000","last_undegraded":"2024-04-16T07:43:00.103138+0000","last_fullsized":"2024-04-16T07:43:00.103138+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1507328,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.426126+0000","last_change":"2024-04-16T07:42:45.334636+0000","last_active":"2024-04-16T07:42:58.426126+0000","last_peered":"2024-04-16T07:42:58.426126+0000","last_clean":"2024-04-16T07:42:58.426126+0000","last_became_active":"2024-04-16T07:42:45.329643+0000","last_became_peered":"2024-04-16T07:42:45.329643+0000","last_unstale":"2024-04-16T07:42:58.426126+0000","last_undegraded":"2024-04-16T07:42:58.426126+0000","last_fullsized":"2024-04-16T07:42:58.426126+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"17'24","reported_seq":37,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.036845+0000","last_change":"2024-04-16T07:42:57.533906+0000","last_active":"2024-04-16T07:43:00.036845+0000","last_peered":"2024-04-16T07:43:00.036845+0000","last_clean":"2024-04-16T07:43:00.036845+0000","last_became_active":"2024-04-16T07:42:57.527972+0000","last_became_peered":"2024-04-16T07:42:57.527972+0000","last_unstale":"2024-04-16T07:43:00.036845+0000","last_undegraded":"2024-04-16T07:43:00.036845+0000","last_fullsized":"2024-04-16T07:43:00.036845+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.317746+0000","last_change":"2024-04-16T07:42:45.327681+0000","last_active":"2024-04-16T07:42:58.317746+0000","last_peered":"2024-04-16T07:42:58.317746+0000","last_clean":"2024-04-16T07:42:58.317746+0000","last_became_active":"2024-04-16T07:42:45.318385+0000","last_became_peered":"2024-04-16T07:42:45.318385+0000","last_unstale":"2024-04-16T07:42:58.317746+0000","last_undegraded":"2024-04-16T07:42:58.317746+0000","last_fullsized":"2024-04-16T07:42:58.317746+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'20","reported_seq":33,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.206863+0000","last_change":"2024-04-16T07:42:57.487393+0000","last_active":"2024-04-16T07:43:00.206863+0000","last_peered":"2024-04-16T07:43:00.206863+0000","last_clean":"2024-04-16T07:43:00.206863+0000","last_became_active":"2024-04-16T07:42:57.481110+0000","last_became_peered":"2024-04-16T07:42:57.481110+0000","last_unstale":"2024-04-16T07:43:00.206863+0000","last_undegraded":"2024-04-16T07:43:00.206863+0000","last_fullsized":"2024-04-16T07:43:00.206863+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1376256,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":1344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.419174+0000","last_change":"2024-04-16T07:42:45.357843+0000","last_active":"2024-04-16T07:42:58.419174+0000","last_peered":"2024-04-16T07:42:58.419174+0000","last_clean":"2024-04-16T07:42:58.419174+0000","last_became_active":"2024-04-16T07:42:45.348908+0000","last_became_peered":"2024-04-16T07:42:45.348908+0000","last_unstale":"2024-04-16T07:42:58.419174+0000","last_undegraded":"2024-04-16T07:42:58.419174+0000","last_fullsized":"2024-04-16T07:42:58.419174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"17'14","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:59.973743+0000","last_change":"2024-04-16T07:42:57.437012+0000","last_active":"2024-04-16T07:42:59.973743+0000","last_peered":"2024-04-16T07:42:59.973743+0000","last_clean":"2024-04-16T07:42:59.973743+0000","last_became_active":"2024-04-16T07:42:57.429704+0000","last_became_peered":"2024-04-16T07:42:57.429704+0000","last_unstale":"2024-04-16T07:42:59.973743+0000","last_undegraded":"2024-04-16T07:42:59.973743+0000","last_fullsized":"2024-04-16T07:42:59.973743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":983040,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'22","reported_seq":35,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.152803+0000","last_change":"2024-04-16T07:42:57.463631+0000","last_active":"2024-04-16T07:43:00.152803+0000","last_peered":"2024-04-16T07:43:00.152803+0000","last_clean":"2024-04-16T07:43:00.152803+0000","last_became_active":"2024-04-16T07:42:57.453387+0000","last_became_peered":"2024-04-16T07:42:57.453387+0000","last_unstale":"2024-04-16T07:43:00.152803+0000","last_undegraded":"2024-04-16T07:43:00.152803+0000","last_fullsized":"2024-04-16T07:43:00.152803+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1507328,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.434405+0000","last_change":"2024-04-16T07:42:45.340605+0000","last_active":"2024-04-16T07:42:58.434405+0000","last_peered":"2024-04-16T07:42:58.434405+0000","last_clean":"2024-04-16T07:42:58.434405+0000","last_became_active":"2024-04-16T07:42:45.335238+0000","last_became_peered":"2024-04-16T07:42:45.335238+0000","last_unstale":"2024-04-16T07:42:58.434405+0000","last_undegraded":"2024-04-16T07:42:58.434405+0000","last_fullsized":"2024-04-16T07:42:58.434405+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"17'28","reported_seq":41,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.229591+0000","last_change":"2024-04-16T07:42:57.470835+0000","last_active":"2024-04-16T07:43:00.229591+0000","last_peered":"2024-04-16T07:43:00.229591+0000","last_clean":"2024-04-16T07:43:00.229591+0000","last_became_active":"2024-04-16T07:42:57.460356+0000","last_became_peered":"2024-04-16T07:42:57.460356+0000","last_unstale":"2024-04-16T07:43:00.229591+0000","last_undegraded":"2024-04-16T07:43:00.229591+0000","last_fullsized":"2024-04-16T07:43:00.229591+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.430163+0000","last_change":"2024-04-16T07:42:45.332152+0000","last_active":"2024-04-16T07:42:58.430163+0000","last_peered":"2024-04-16T07:42:58.430163+0000","last_clean":"2024-04-16T07:42:58.430163+0000","last_became_active":"2024-04-16T07:42:45.325833+0000","last_became_peered":"2024-04-16T07:42:45.325833+0000","last_unstale":"2024-04-16T07:42:58.430163+0000","last_undegraded":"2024-04-16T07:42:58.430163+0000","last_fullsized":"2024-04-16T07:42:58.430163+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'29","reported_seq":42,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.157835+0000","last_change":"2024-04-16T07:42:57.489443+0000","last_active":"2024-04-16T07:43:00.157835+0000","last_peered":"2024-04-16T07:43:00.157835+0000","last_clean":"2024-04-16T07:43:00.157835+0000","last_became_active":"2024-04-16T07:42:57.480103+0000","last_became_peered":"2024-04-16T07:42:57.480103+0000","last_unstale":"2024-04-16T07:43:00.157835+0000","last_undegraded":"2024-04-16T07:43:00.157835+0000","last_fullsized":"2024-04-16T07:43:00.157835+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1966080,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":1920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.410936+0000","last_change":"2024-04-16T07:42:45.333004+0000","last_active":"2024-04-16T07:42:58.410936+0000","last_peered":"2024-04-16T07:42:58.410936+0000","last_clean":"2024-04-16T07:42:58.410936+0000","last_became_active":"2024-04-16T07:42:45.323737+0000","last_became_peered":"2024-04-16T07:42:45.323737+0000","last_unstale":"2024-04-16T07:42:58.410936+0000","last_undegraded":"2024-04-16T07:42:58.410936+0000","last_fullsized":"2024-04-16T07:42:58.410936+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'21","reported_seq":34,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.216750+0000","last_change":"2024-04-16T07:42:57.475347+0000","last_active":"2024-04-16T07:43:00.216750+0000","last_peered":"2024-04-16T07:43:00.216750+0000","last_clean":"2024-04-16T07:43:00.216750+0000","last_became_active":"2024-04-16T07:42:57.465803+0000","last_became_peered":"2024-04-16T07:42:57.465803+0000","last_unstale":"2024-04-16T07:43:00.216750+0000","last_undegraded":"2024-04-16T07:43:00.216750+0000","last_fullsized":"2024-04-16T07:43:00.216750+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.329672+0000","last_change":"2024-04-16T07:42:45.376121+0000","last_active":"2024-04-16T07:42:58.329672+0000","last_peered":"2024-04-16T07:42:58.329672+0000","last_clean":"2024-04-16T07:42:58.329672+0000","last_became_active":"2024-04-16T07:42:45.370390+0000","last_became_peered":"2024-04-16T07:42:45.370390+0000","last_unstale":"2024-04-16T07:42:58.329672+0000","last_undegraded":"2024-04-16T07:42:58.329672+0000","last_fullsized":"2024-04-16T07:42:58.329672+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":25362432,"num_objects":387,"num_object_clones":0,"num_object_copies":1161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":387,"num_write_kb":24768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":371,"ondisk_log_size":371,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705684,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27608,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":802816,"data_stored":10199845,"data_compressed":484060,"data_compressed_allocated":638976,"data_compressed_original":10141696,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705683,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27456,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":647168,"data_stored":7709477,"data_compressed":365874,"data_compressed_allocated":483328,"data_compressed_original":7651328,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705683,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27504,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":696320,"data_stored":8888581,"data_compressed":419892,"data_compressed_allocated":552960,"data_compressed_original":8847360,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705683,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27524,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344316,"statfs":{"total":96636764160,"available":96608579584,"internally_reserved":0,"allocated":716800,"data_stored":8823589,"data_compressed":418754,"data_compressed_allocated":552960,"data_compressed_original":8765440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:43:01.392 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 3 16 635 619 12.8947 13.1875 0.0773988 0.0762175 2024-04-16T07:43:02.392 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 4 16 858 842 13.1551 13.9375 0.0404302 0.0751206 2024-04-16T07:43:02.515 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:43:02.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:02.680+0000 7f31650ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:02.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:02.690+0000 7f31650ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:03.392 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 5 16 1077 1061 13.2613 13.6875 0.0449536 0.074979 2024-04-16T07:43:04.392 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 6 16 1299 1283 13.3634 13.875 0.0855025 0.0743733 2024-04-16T07:43:05.107 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:43:05.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:05.250+0000 7f6b810ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:05.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:05.258+0000 7f6b810ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:05.392 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 7 16 1495 1479 13.2041 12.25 0.125718 0.0752128 2024-04-16T07:43:06.295 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-16T07:43:06.295 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:43:06.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:43:06.392 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 8 16 1698 1682 13.1394 12.6875 0.0806156 0.0756729 2024-04-16T07:43:06.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:06.452+0000 7f9cb426d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:06.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:06.467+0000 7f9cb426d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:06.771 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:43:06.771 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-16T07:43:06.791 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-16T07:43:05.220693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25952800,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":524,"num_write_kb":26662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":468,"ondisk_log_size":468,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":110092,"kb_used_data":2796,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377377268,"statfs":{"total":386547056640,"available":386434322432,"internally_reserved":0,"allocated":2863104,"data_stored":35621492,"data_compressed":1688580,"data_compressed_allocated":2228224,"data_compressed_original":35405824,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25362432,"num_objects":387,"num_object_clones":0,"num_object_copies":1161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":387,"num_write_kb":24768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001703"},"pg_stats":[{"pgid":"3.f","version":"17'23","reported_seq":36,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:59.977304+0000","last_change":"2024-04-16T07:42:57.456939+0000","last_active":"2024-04-16T07:42:59.977304+0000","last_peered":"2024-04-16T07:42:59.977304+0000","last_clean":"2024-04-16T07:42:59.977304+0000","last_became_active":"2024-04-16T07:42:57.450056+0000","last_became_peered":"2024-04-16T07:42:57.450056+0000","last_unstale":"2024-04-16T07:42:59.977304+0000","last_undegraded":"2024-04-16T07:42:59.977304+0000","last_fullsized":"2024-04-16T07:42:59.977304+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"17'24","reported_seq":37,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:59.951608+0000","last_change":"2024-04-16T07:42:57.453661+0000","last_active":"2024-04-16T07:42:59.951608+0000","last_peered":"2024-04-16T07:42:59.951608+0000","last_clean":"2024-04-16T07:42:59.951608+0000","last_became_active":"2024-04-16T07:42:57.447818+0000","last_became_peered":"2024-04-16T07:42:57.447818+0000","last_unstale":"2024-04-16T07:42:59.951608+0000","last_undegraded":"2024-04-16T07:42:59.951608+0000","last_fullsized":"2024-04-16T07:42:59.951608+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'26","reported_seq":39,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:59.868565+0000","last_change":"2024-04-16T07:42:57.507834+0000","last_active":"2024-04-16T07:42:59.868565+0000","last_peered":"2024-04-16T07:42:59.868565+0000","last_clean":"2024-04-16T07:42:59.868565+0000","last_became_active":"2024-04-16T07:42:57.499895+0000","last_became_peered":"2024-04-16T07:42:57.499895+0000","last_unstale":"2024-04-16T07:42:59.868565+0000","last_undegraded":"2024-04-16T07:42:59.868565+0000","last_fullsized":"2024-04-16T07:42:59.868565+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'28","reported_seq":41,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.220396+0000","last_change":"2024-04-16T07:42:57.465377+0000","last_active":"2024-04-16T07:43:00.220396+0000","last_peered":"2024-04-16T07:43:00.220396+0000","last_clean":"2024-04-16T07:43:00.220396+0000","last_became_active":"2024-04-16T07:42:57.451654+0000","last_became_peered":"2024-04-16T07:42:57.451654+0000","last_unstale":"2024-04-16T07:43:00.220396+0000","last_undegraded":"2024-04-16T07:43:00.220396+0000","last_fullsized":"2024-04-16T07:43:00.220396+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'26","reported_seq":39,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.019395+0000","last_change":"2024-04-16T07:42:57.459819+0000","last_active":"2024-04-16T07:43:00.019395+0000","last_peered":"2024-04-16T07:43:00.019395+0000","last_clean":"2024-04-16T07:43:00.019395+0000","last_became_active":"2024-04-16T07:42:57.453218+0000","last_became_peered":"2024-04-16T07:42:57.453218+0000","last_unstale":"2024-04-16T07:43:00.019395+0000","last_undegraded":"2024-04-16T07:43:00.019395+0000","last_fullsized":"2024-04-16T07:43:00.019395+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'21","reported_seq":34,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.120346+0000","last_change":"2024-04-16T07:42:57.489933+0000","last_active":"2024-04-16T07:43:00.120346+0000","last_peered":"2024-04-16T07:43:00.120346+0000","last_clean":"2024-04-16T07:43:00.120346+0000","last_became_active":"2024-04-16T07:42:57.482517+0000","last_became_peered":"2024-04-16T07:42:57.482517+0000","last_unstale":"2024-04-16T07:43:00.120346+0000","last_undegraded":"2024-04-16T07:43:00.120346+0000","last_fullsized":"2024-04-16T07:43:00.120346+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'22","reported_seq":35,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.172645+0000","last_change":"2024-04-16T07:42:57.469015+0000","last_active":"2024-04-16T07:43:00.172645+0000","last_peered":"2024-04-16T07:43:00.172645+0000","last_clean":"2024-04-16T07:43:00.172645+0000","last_became_active":"2024-04-16T07:42:57.460537+0000","last_became_peered":"2024-04-16T07:42:57.460537+0000","last_unstale":"2024-04-16T07:43:00.172645+0000","last_undegraded":"2024-04-16T07:43:00.172645+0000","last_fullsized":"2024-04-16T07:43:00.172645+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1507328,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"17'21","reported_seq":34,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.235372+0000","last_change":"2024-04-16T07:42:57.459433+0000","last_active":"2024-04-16T07:43:00.235372+0000","last_peered":"2024-04-16T07:43:00.235372+0000","last_clean":"2024-04-16T07:43:00.235372+0000","last_became_active":"2024-04-16T07:42:57.452612+0000","last_became_peered":"2024-04-16T07:42:57.452612+0000","last_unstale":"2024-04-16T07:43:00.235372+0000","last_undegraded":"2024-04-16T07:43:00.235372+0000","last_fullsized":"2024-04-16T07:43:00.235372+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":154,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.415419+0000","last_change":"2024-04-16T07:42:43.287780+0000","last_active":"2024-04-16T07:42:58.415419+0000","last_peered":"2024-04-16T07:42:58.415419+0000","last_clean":"2024-04-16T07:42:58.415419+0000","last_became_active":"2024-04-16T07:42:43.282375+0000","last_became_peered":"2024-04-16T07:42:43.282375+0000","last_unstale":"2024-04-16T07:42:58.415419+0000","last_undegraded":"2024-04-16T07:42:58.415419+0000","last_fullsized":"2024-04-16T07:42:58.415419+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_clean_scrub_stamp":"2024-04-16T07:42:42.229959+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.331433+0000","last_change":"2024-04-16T07:42:45.350220+0000","last_active":"2024-04-16T07:42:58.331433+0000","last_peered":"2024-04-16T07:42:58.331433+0000","last_clean":"2024-04-16T07:42:58.331433+0000","last_became_active":"2024-04-16T07:42:45.343569+0000","last_became_peered":"2024-04-16T07:42:45.343569+0000","last_unstale":"2024-04-16T07:42:58.331433+0000","last_undegraded":"2024-04-16T07:42:58.331433+0000","last_fullsized":"2024-04-16T07:42:58.331433+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"17'22","reported_seq":35,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.103138+0000","last_change":"2024-04-16T07:42:57.476764+0000","last_active":"2024-04-16T07:43:00.103138+0000","last_peered":"2024-04-16T07:43:00.103138+0000","last_clean":"2024-04-16T07:43:00.103138+0000","last_became_active":"2024-04-16T07:42:57.471552+0000","last_became_peered":"2024-04-16T07:42:57.471552+0000","last_unstale":"2024-04-16T07:43:00.103138+0000","last_undegraded":"2024-04-16T07:43:00.103138+0000","last_fullsized":"2024-04-16T07:43:00.103138+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1507328,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.426126+0000","last_change":"2024-04-16T07:42:45.334636+0000","last_active":"2024-04-16T07:42:58.426126+0000","last_peered":"2024-04-16T07:42:58.426126+0000","last_clean":"2024-04-16T07:42:58.426126+0000","last_became_active":"2024-04-16T07:42:45.329643+0000","last_became_peered":"2024-04-16T07:42:45.329643+0000","last_unstale":"2024-04-16T07:42:58.426126+0000","last_undegraded":"2024-04-16T07:42:58.426126+0000","last_fullsized":"2024-04-16T07:42:58.426126+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"17'24","reported_seq":37,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.036845+0000","last_change":"2024-04-16T07:42:57.533906+0000","last_active":"2024-04-16T07:43:00.036845+0000","last_peered":"2024-04-16T07:43:00.036845+0000","last_clean":"2024-04-16T07:43:00.036845+0000","last_became_active":"2024-04-16T07:42:57.527972+0000","last_became_peered":"2024-04-16T07:42:57.527972+0000","last_unstale":"2024-04-16T07:43:00.036845+0000","last_undegraded":"2024-04-16T07:43:00.036845+0000","last_fullsized":"2024-04-16T07:43:00.036845+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.317746+0000","last_change":"2024-04-16T07:42:45.327681+0000","last_active":"2024-04-16T07:42:58.317746+0000","last_peered":"2024-04-16T07:42:58.317746+0000","last_clean":"2024-04-16T07:42:58.317746+0000","last_became_active":"2024-04-16T07:42:45.318385+0000","last_became_peered":"2024-04-16T07:42:45.318385+0000","last_unstale":"2024-04-16T07:42:58.317746+0000","last_undegraded":"2024-04-16T07:42:58.317746+0000","last_fullsized":"2024-04-16T07:42:58.317746+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'20","reported_seq":33,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.206863+0000","last_change":"2024-04-16T07:42:57.487393+0000","last_active":"2024-04-16T07:43:00.206863+0000","last_peered":"2024-04-16T07:43:00.206863+0000","last_clean":"2024-04-16T07:43:00.206863+0000","last_became_active":"2024-04-16T07:42:57.481110+0000","last_became_peered":"2024-04-16T07:42:57.481110+0000","last_unstale":"2024-04-16T07:43:00.206863+0000","last_undegraded":"2024-04-16T07:43:00.206863+0000","last_fullsized":"2024-04-16T07:43:00.206863+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1376256,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":1344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.419174+0000","last_change":"2024-04-16T07:42:45.357843+0000","last_active":"2024-04-16T07:42:58.419174+0000","last_peered":"2024-04-16T07:42:58.419174+0000","last_clean":"2024-04-16T07:42:58.419174+0000","last_became_active":"2024-04-16T07:42:45.348908+0000","last_became_peered":"2024-04-16T07:42:45.348908+0000","last_unstale":"2024-04-16T07:42:58.419174+0000","last_undegraded":"2024-04-16T07:42:58.419174+0000","last_fullsized":"2024-04-16T07:42:58.419174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"17'14","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:59.973743+0000","last_change":"2024-04-16T07:42:57.437012+0000","last_active":"2024-04-16T07:42:59.973743+0000","last_peered":"2024-04-16T07:42:59.973743+0000","last_clean":"2024-04-16T07:42:59.973743+0000","last_became_active":"2024-04-16T07:42:57.429704+0000","last_became_peered":"2024-04-16T07:42:57.429704+0000","last_unstale":"2024-04-16T07:42:59.973743+0000","last_undegraded":"2024-04-16T07:42:59.973743+0000","last_fullsized":"2024-04-16T07:42:59.973743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":983040,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'22","reported_seq":35,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.152803+0000","last_change":"2024-04-16T07:42:57.463631+0000","last_active":"2024-04-16T07:43:00.152803+0000","last_peered":"2024-04-16T07:43:00.152803+0000","last_clean":"2024-04-16T07:43:00.152803+0000","last_became_active":"2024-04-16T07:42:57.453387+0000","last_became_peered":"2024-04-16T07:42:57.453387+0000","last_unstale":"2024-04-16T07:43:00.152803+0000","last_undegraded":"2024-04-16T07:43:00.152803+0000","last_fullsized":"2024-04-16T07:43:00.152803+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1507328,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.434405+0000","last_change":"2024-04-16T07:42:45.340605+0000","last_active":"2024-04-16T07:42:58.434405+0000","last_peered":"2024-04-16T07:42:58.434405+0000","last_clean":"2024-04-16T07:42:58.434405+0000","last_became_active":"2024-04-16T07:42:45.335238+0000","last_became_peered":"2024-04-16T07:42:45.335238+0000","last_unstale":"2024-04-16T07:42:58.434405+0000","last_undegraded":"2024-04-16T07:42:58.434405+0000","last_fullsized":"2024-04-16T07:42:58.434405+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"17'28","reported_seq":41,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.229591+0000","last_change":"2024-04-16T07:42:57.470835+0000","last_active":"2024-04-16T07:43:00.229591+0000","last_peered":"2024-04-16T07:43:00.229591+0000","last_clean":"2024-04-16T07:43:00.229591+0000","last_became_active":"2024-04-16T07:42:57.460356+0000","last_became_peered":"2024-04-16T07:42:57.460356+0000","last_unstale":"2024-04-16T07:43:00.229591+0000","last_undegraded":"2024-04-16T07:43:00.229591+0000","last_fullsized":"2024-04-16T07:43:00.229591+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.430163+0000","last_change":"2024-04-16T07:42:45.332152+0000","last_active":"2024-04-16T07:42:58.430163+0000","last_peered":"2024-04-16T07:42:58.430163+0000","last_clean":"2024-04-16T07:42:58.430163+0000","last_became_active":"2024-04-16T07:42:45.325833+0000","last_became_peered":"2024-04-16T07:42:45.325833+0000","last_unstale":"2024-04-16T07:42:58.430163+0000","last_undegraded":"2024-04-16T07:42:58.430163+0000","last_fullsized":"2024-04-16T07:42:58.430163+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'29","reported_seq":42,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.157835+0000","last_change":"2024-04-16T07:42:57.489443+0000","last_active":"2024-04-16T07:43:00.157835+0000","last_peered":"2024-04-16T07:43:00.157835+0000","last_clean":"2024-04-16T07:43:00.157835+0000","last_became_active":"2024-04-16T07:42:57.480103+0000","last_became_peered":"2024-04-16T07:42:57.480103+0000","last_unstale":"2024-04-16T07:43:00.157835+0000","last_undegraded":"2024-04-16T07:43:00.157835+0000","last_fullsized":"2024-04-16T07:43:00.157835+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1966080,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":1920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.410936+0000","last_change":"2024-04-16T07:42:45.333004+0000","last_active":"2024-04-16T07:42:58.410936+0000","last_peered":"2024-04-16T07:42:58.410936+0000","last_clean":"2024-04-16T07:42:58.410936+0000","last_became_active":"2024-04-16T07:42:45.323737+0000","last_became_peered":"2024-04-16T07:42:45.323737+0000","last_unstale":"2024-04-16T07:42:58.410936+0000","last_undegraded":"2024-04-16T07:42:58.410936+0000","last_fullsized":"2024-04-16T07:42:58.410936+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'21","reported_seq":34,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:00.216750+0000","last_change":"2024-04-16T07:42:57.475347+0000","last_active":"2024-04-16T07:43:00.216750+0000","last_peered":"2024-04-16T07:43:00.216750+0000","last_clean":"2024-04-16T07:43:00.216750+0000","last_became_active":"2024-04-16T07:42:57.465803+0000","last_became_peered":"2024-04-16T07:42:57.465803+0000","last_unstale":"2024-04-16T07:43:00.216750+0000","last_undegraded":"2024-04-16T07:43:00.216750+0000","last_fullsized":"2024-04-16T07:43:00.216750+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.329672+0000","last_change":"2024-04-16T07:42:45.376121+0000","last_active":"2024-04-16T07:42:58.329672+0000","last_peered":"2024-04-16T07:42:58.329672+0000","last_clean":"2024-04-16T07:42:58.329672+0000","last_became_active":"2024-04-16T07:42:45.370390+0000","last_became_peered":"2024-04-16T07:42:45.370390+0000","last_unstale":"2024-04-16T07:42:58.329672+0000","last_undegraded":"2024-04-16T07:42:58.329672+0000","last_fullsized":"2024-04-16T07:42:58.329672+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":25362432,"num_objects":387,"num_object_clones":0,"num_object_copies":1161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":387,"num_write_kb":24768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":371,"ondisk_log_size":371,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705684,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27608,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":802816,"data_stored":10199845,"data_compressed":484060,"data_compressed_allocated":638976,"data_compressed_original":10141696,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705683,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27456,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":647168,"data_stored":7709477,"data_compressed":365874,"data_compressed_allocated":483328,"data_compressed_original":7651328,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705683,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27504,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":696320,"data_stored":8888581,"data_compressed":419892,"data_compressed_allocated":552960,"data_compressed_original":8847360,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705683,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27524,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344316,"statfs":{"total":96636764160,"available":96608579584,"internally_reserved":0,"allocated":716800,"data_stored":8823589,"data_compressed":418754,"data_compressed_allocated":552960,"data_compressed_original":8765440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:43:07.393 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 9 16 1902 1886 13.096 12.75 0.0451423 0.0759 2024-04-16T07:43:07.597 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:43:07.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:07.747+0000 7f2641d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:07.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:07.757+0000 7f2641d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:08.393 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 10 16 2121 2105 13.1551 13.6875 0.0536765 0.0756478 2024-04-16T07:43:09.393 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 11 16 2327 2311 13.1295 12.875 0.0914487 0.0757707 2024-04-16T07:43:10.214 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:43:10.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:10.371+0000 7f696faad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:10.380+0000 7f696faad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:10.393 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 12 16 2541 2525 13.1498 13.375 0.0929439 0.0758057 2024-04-16T07:43:11.393 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 13 16 2743 2727 13.1094 12.625 0.116855 0.0760668 2024-04-16T07:43:11.793 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-16T07:43:11.793 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:43:11.793 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:43:11.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:11.958+0000 7f35c0aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:11.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:11.973+0000 7f35c0aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:12.349 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:43:12.349 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-16T07:43:12.368 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-16T07:43:11.221987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":165020192,"num_objects":2511,"num_object_clones":0,"num_object_copies":7533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":2646,"num_write_kb":162470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2590,"ondisk_log_size":2590,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":156432,"kb_used_data":28336,"kb_used_omap":0,"kb_used_meta":128064,"kb_avail":377330928,"statfs":{"total":386547056640,"available":386386870272,"internally_reserved":0,"allocated":29016064,"data_stored":454068852,"data_compressed":21556214,"data_compressed_allocated":28381184,"data_compressed_original":453853184,"omap_allocated":0,"internal_metadata":131137536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":164429824,"num_objects":2509,"num_object_clones":0,"num_object_copies":7527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2509,"num_write_kb":160576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002637"},"pg_stats":[{"pgid":"3.f","version":"17'146","reported_seq":159,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:10.098620+0000","last_change":"2024-04-16T07:42:57.456939+0000","last_active":"2024-04-16T07:43:10.098620+0000","last_peered":"2024-04-16T07:43:10.098620+0000","last_clean":"2024-04-16T07:43:10.098620+0000","last_became_active":"2024-04-16T07:42:57.450056+0000","last_became_peered":"2024-04-16T07:42:57.450056+0000","last_unstale":"2024-04-16T07:43:10.098620+0000","last_undegraded":"2024-04-16T07:43:10.098620+0000","last_fullsized":"2024-04-16T07:43:10.098620+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9633792,"num_objects":147,"num_object_clones":0,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"17'146","reported_seq":159,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:09.983009+0000","last_change":"2024-04-16T07:42:57.453661+0000","last_active":"2024-04-16T07:43:09.983009+0000","last_peered":"2024-04-16T07:43:09.983009+0000","last_clean":"2024-04-16T07:43:09.983009+0000","last_became_active":"2024-04-16T07:42:57.447818+0000","last_became_peered":"2024-04-16T07:42:57.447818+0000","last_unstale":"2024-04-16T07:43:09.983009+0000","last_undegraded":"2024-04-16T07:43:09.983009+0000","last_fullsized":"2024-04-16T07:43:09.983009+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9633792,"num_objects":147,"num_object_clones":0,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'149","reported_seq":162,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:09.990381+0000","last_change":"2024-04-16T07:42:57.507834+0000","last_active":"2024-04-16T07:43:09.990381+0000","last_peered":"2024-04-16T07:43:09.990381+0000","last_clean":"2024-04-16T07:43:09.990381+0000","last_became_active":"2024-04-16T07:42:57.499895+0000","last_became_peered":"2024-04-16T07:42:57.499895+0000","last_unstale":"2024-04-16T07:43:09.990381+0000","last_undegraded":"2024-04-16T07:43:09.990381+0000","last_fullsized":"2024-04-16T07:43:09.990381+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9830400,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":9600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'153","reported_seq":166,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:10.208941+0000","last_change":"2024-04-16T07:42:57.465377+0000","last_active":"2024-04-16T07:43:10.208941+0000","last_peered":"2024-04-16T07:43:10.208941+0000","last_clean":"2024-04-16T07:43:10.208941+0000","last_became_active":"2024-04-16T07:42:57.451654+0000","last_became_peered":"2024-04-16T07:42:57.451654+0000","last_unstale":"2024-04-16T07:43:10.208941+0000","last_undegraded":"2024-04-16T07:43:10.208941+0000","last_fullsized":"2024-04-16T07:43:10.208941+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10092544,"num_objects":154,"num_object_clones":0,"num_object_copies":462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154,"num_write_kb":9856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'148","reported_seq":161,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:10.172556+0000","last_change":"2024-04-16T07:42:57.459819+0000","last_active":"2024-04-16T07:43:10.172556+0000","last_peered":"2024-04-16T07:43:10.172556+0000","last_clean":"2024-04-16T07:43:10.172556+0000","last_became_active":"2024-04-16T07:42:57.453218+0000","last_became_peered":"2024-04-16T07:42:57.453218+0000","last_unstale":"2024-04-16T07:43:10.172556+0000","last_undegraded":"2024-04-16T07:43:10.172556+0000","last_fullsized":"2024-04-16T07:43:10.172556+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9764864,"num_objects":149,"num_object_clones":0,"num_object_copies":447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":149,"num_write_kb":9536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'168","reported_seq":181,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:10.180290+0000","last_change":"2024-04-16T07:42:57.489933+0000","last_active":"2024-04-16T07:43:10.180290+0000","last_peered":"2024-04-16T07:43:10.180290+0000","last_clean":"2024-04-16T07:43:10.180290+0000","last_became_active":"2024-04-16T07:42:57.482517+0000","last_became_peered":"2024-04-16T07:42:57.482517+0000","last_unstale":"2024-04-16T07:43:10.180290+0000","last_undegraded":"2024-04-16T07:43:10.180290+0000","last_fullsized":"2024-04-16T07:43:10.180290+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11075584,"num_objects":169,"num_object_clones":0,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":10816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'153","reported_seq":166,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:10.234743+0000","last_change":"2024-04-16T07:42:57.469015+0000","last_active":"2024-04-16T07:43:10.234743+0000","last_peered":"2024-04-16T07:43:10.234743+0000","last_clean":"2024-04-16T07:43:10.234743+0000","last_became_active":"2024-04-16T07:42:57.460537+0000","last_became_peered":"2024-04-16T07:42:57.460537+0000","last_unstale":"2024-04-16T07:43:10.234743+0000","last_undegraded":"2024-04-16T07:43:10.234743+0000","last_fullsized":"2024-04-16T07:43:10.234743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10092544,"num_objects":154,"num_object_clones":0,"num_object_copies":462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154,"num_write_kb":9856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"17'160","reported_seq":173,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:10.152374+0000","last_change":"2024-04-16T07:42:57.459433+0000","last_active":"2024-04-16T07:43:10.152374+0000","last_peered":"2024-04-16T07:43:10.152374+0000","last_clean":"2024-04-16T07:43:10.152374+0000","last_became_active":"2024-04-16T07:42:57.452612+0000","last_became_peered":"2024-04-16T07:42:57.452612+0000","last_unstale":"2024-04-16T07:43:10.152374+0000","last_undegraded":"2024-04-16T07:43:10.152374+0000","last_fullsized":"2024-04-16T07:43:10.152374+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10551296,"num_objects":161,"num_object_clones":0,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":161,"num_write_kb":10304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":154,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.415419+0000","last_change":"2024-04-16T07:42:43.287780+0000","last_active":"2024-04-16T07:42:58.415419+0000","last_peered":"2024-04-16T07:42:58.415419+0000","last_clean":"2024-04-16T07:42:58.415419+0000","last_became_active":"2024-04-16T07:42:43.282375+0000","last_became_peered":"2024-04-16T07:42:43.282375+0000","last_unstale":"2024-04-16T07:42:58.415419+0000","last_undegraded":"2024-04-16T07:42:58.415419+0000","last_fullsized":"2024-04-16T07:42:58.415419+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_clean_scrub_stamp":"2024-04-16T07:42:42.229959+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.331433+0000","last_change":"2024-04-16T07:42:45.350220+0000","last_active":"2024-04-16T07:42:58.331433+0000","last_peered":"2024-04-16T07:42:58.331433+0000","last_clean":"2024-04-16T07:42:58.331433+0000","last_became_active":"2024-04-16T07:42:45.343569+0000","last_became_peered":"2024-04-16T07:42:45.343569+0000","last_unstale":"2024-04-16T07:42:58.331433+0000","last_undegraded":"2024-04-16T07:42:58.331433+0000","last_fullsized":"2024-04-16T07:42:58.331433+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"17'170","reported_seq":183,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:10.148233+0000","last_change":"2024-04-16T07:42:57.476764+0000","last_active":"2024-04-16T07:43:10.148233+0000","last_peered":"2024-04-16T07:43:10.148233+0000","last_clean":"2024-04-16T07:43:10.148233+0000","last_became_active":"2024-04-16T07:42:57.471552+0000","last_became_peered":"2024-04-16T07:42:57.471552+0000","last_unstale":"2024-04-16T07:43:10.148233+0000","last_undegraded":"2024-04-16T07:43:10.148233+0000","last_fullsized":"2024-04-16T07:43:10.148233+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11206656,"num_objects":171,"num_object_clones":0,"num_object_copies":513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":171,"num_write_kb":10944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.426126+0000","last_change":"2024-04-16T07:42:45.334636+0000","last_active":"2024-04-16T07:42:58.426126+0000","last_peered":"2024-04-16T07:42:58.426126+0000","last_clean":"2024-04-16T07:42:58.426126+0000","last_became_active":"2024-04-16T07:42:45.329643+0000","last_became_peered":"2024-04-16T07:42:45.329643+0000","last_unstale":"2024-04-16T07:42:58.426126+0000","last_undegraded":"2024-04-16T07:42:58.426126+0000","last_fullsized":"2024-04-16T07:42:58.426126+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"17'170","reported_seq":183,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:10.054037+0000","last_change":"2024-04-16T07:42:57.533906+0000","last_active":"2024-04-16T07:43:10.054037+0000","last_peered":"2024-04-16T07:43:10.054037+0000","last_clean":"2024-04-16T07:43:10.054037+0000","last_became_active":"2024-04-16T07:42:57.527972+0000","last_became_peered":"2024-04-16T07:42:57.527972+0000","last_unstale":"2024-04-16T07:43:10.054037+0000","last_undegraded":"2024-04-16T07:43:10.054037+0000","last_fullsized":"2024-04-16T07:43:10.054037+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11206656,"num_objects":171,"num_object_clones":0,"num_object_copies":513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":171,"num_write_kb":10944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.317746+0000","last_change":"2024-04-16T07:42:45.327681+0000","last_active":"2024-04-16T07:42:58.317746+0000","last_peered":"2024-04-16T07:42:58.317746+0000","last_clean":"2024-04-16T07:42:58.317746+0000","last_became_active":"2024-04-16T07:42:45.318385+0000","last_became_peered":"2024-04-16T07:42:45.318385+0000","last_unstale":"2024-04-16T07:42:58.317746+0000","last_undegraded":"2024-04-16T07:42:58.317746+0000","last_fullsized":"2024-04-16T07:42:58.317746+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'150","reported_seq":163,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:10.046543+0000","last_change":"2024-04-16T07:42:57.487393+0000","last_active":"2024-04-16T07:43:10.046543+0000","last_peered":"2024-04-16T07:43:10.046543+0000","last_clean":"2024-04-16T07:43:10.046543+0000","last_became_active":"2024-04-16T07:42:57.481110+0000","last_became_peered":"2024-04-16T07:42:57.481110+0000","last_unstale":"2024-04-16T07:43:10.046543+0000","last_undegraded":"2024-04-16T07:43:10.046543+0000","last_fullsized":"2024-04-16T07:43:10.046543+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9895936,"num_objects":151,"num_object_clones":0,"num_object_copies":453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":151,"num_write_kb":9664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.419174+0000","last_change":"2024-04-16T07:42:45.357843+0000","last_active":"2024-04-16T07:42:58.419174+0000","last_peered":"2024-04-16T07:42:58.419174+0000","last_clean":"2024-04-16T07:42:58.419174+0000","last_became_active":"2024-04-16T07:42:45.348908+0000","last_became_peered":"2024-04-16T07:42:45.348908+0000","last_unstale":"2024-04-16T07:42:58.419174+0000","last_undegraded":"2024-04-16T07:42:58.419174+0000","last_fullsized":"2024-04-16T07:42:58.419174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"17'152","reported_seq":165,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:10.072851+0000","last_change":"2024-04-16T07:42:57.437012+0000","last_active":"2024-04-16T07:43:10.072851+0000","last_peered":"2024-04-16T07:43:10.072851+0000","last_clean":"2024-04-16T07:43:10.072851+0000","last_became_active":"2024-04-16T07:42:57.429704+0000","last_became_peered":"2024-04-16T07:42:57.429704+0000","last_unstale":"2024-04-16T07:43:10.072851+0000","last_undegraded":"2024-04-16T07:43:10.072851+0000","last_fullsized":"2024-04-16T07:43:10.072851+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10027008,"num_objects":153,"num_object_clones":0,"num_object_copies":459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":9792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'162","reported_seq":175,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:09.993828+0000","last_change":"2024-04-16T07:42:57.463631+0000","last_active":"2024-04-16T07:43:09.993828+0000","last_peered":"2024-04-16T07:43:09.993828+0000","last_clean":"2024-04-16T07:43:09.993828+0000","last_became_active":"2024-04-16T07:42:57.453387+0000","last_became_peered":"2024-04-16T07:42:57.453387+0000","last_unstale":"2024-04-16T07:43:09.993828+0000","last_undegraded":"2024-04-16T07:43:09.993828+0000","last_fullsized":"2024-04-16T07:43:09.993828+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10682368,"num_objects":163,"num_object_clones":0,"num_object_copies":489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":163,"num_write_kb":10432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.434405+0000","last_change":"2024-04-16T07:42:45.340605+0000","last_active":"2024-04-16T07:42:58.434405+0000","last_peered":"2024-04-16T07:42:58.434405+0000","last_clean":"2024-04-16T07:42:58.434405+0000","last_became_active":"2024-04-16T07:42:45.335238+0000","last_became_peered":"2024-04-16T07:42:45.335238+0000","last_unstale":"2024-04-16T07:42:58.434405+0000","last_undegraded":"2024-04-16T07:42:58.434405+0000","last_fullsized":"2024-04-16T07:42:58.434405+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"17'153","reported_seq":166,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:10.063599+0000","last_change":"2024-04-16T07:42:57.470835+0000","last_active":"2024-04-16T07:43:10.063599+0000","last_peered":"2024-04-16T07:43:10.063599+0000","last_clean":"2024-04-16T07:43:10.063599+0000","last_became_active":"2024-04-16T07:42:57.460356+0000","last_became_peered":"2024-04-16T07:42:57.460356+0000","last_unstale":"2024-04-16T07:43:10.063599+0000","last_undegraded":"2024-04-16T07:43:10.063599+0000","last_fullsized":"2024-04-16T07:43:10.063599+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10092544,"num_objects":154,"num_object_clones":0,"num_object_copies":462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154,"num_write_kb":9856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.430163+0000","last_change":"2024-04-16T07:42:45.332152+0000","last_active":"2024-04-16T07:42:58.430163+0000","last_peered":"2024-04-16T07:42:58.430163+0000","last_clean":"2024-04-16T07:42:58.430163+0000","last_became_active":"2024-04-16T07:42:45.325833+0000","last_became_peered":"2024-04-16T07:42:45.325833+0000","last_unstale":"2024-04-16T07:42:58.430163+0000","last_undegraded":"2024-04-16T07:42:58.430163+0000","last_fullsized":"2024-04-16T07:42:58.430163+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'172","reported_seq":185,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:10.230741+0000","last_change":"2024-04-16T07:42:57.489443+0000","last_active":"2024-04-16T07:43:10.230741+0000","last_peered":"2024-04-16T07:43:10.230741+0000","last_clean":"2024-04-16T07:43:10.230741+0000","last_became_active":"2024-04-16T07:42:57.480103+0000","last_became_peered":"2024-04-16T07:42:57.480103+0000","last_unstale":"2024-04-16T07:43:10.230741+0000","last_undegraded":"2024-04-16T07:43:10.230741+0000","last_fullsized":"2024-04-16T07:43:10.230741+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11337728,"num_objects":173,"num_object_clones":0,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":173,"num_write_kb":11072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.410936+0000","last_change":"2024-04-16T07:42:45.333004+0000","last_active":"2024-04-16T07:42:58.410936+0000","last_peered":"2024-04-16T07:42:58.410936+0000","last_clean":"2024-04-16T07:42:58.410936+0000","last_became_active":"2024-04-16T07:42:45.323737+0000","last_became_peered":"2024-04-16T07:42:45.323737+0000","last_unstale":"2024-04-16T07:42:58.410936+0000","last_undegraded":"2024-04-16T07:42:58.410936+0000","last_fullsized":"2024-04-16T07:42:58.410936+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'141","reported_seq":154,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:10.124250+0000","last_change":"2024-04-16T07:42:57.475347+0000","last_active":"2024-04-16T07:43:10.124250+0000","last_peered":"2024-04-16T07:43:10.124250+0000","last_clean":"2024-04-16T07:43:10.124250+0000","last_became_active":"2024-04-16T07:42:57.465803+0000","last_became_peered":"2024-04-16T07:42:57.465803+0000","last_unstale":"2024-04-16T07:43:10.124250+0000","last_undegraded":"2024-04-16T07:43:10.124250+0000","last_fullsized":"2024-04-16T07:43:10.124250+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9306112,"num_objects":142,"num_object_clones":0,"num_object_copies":426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":142,"num_write_kb":9088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.329672+0000","last_change":"2024-04-16T07:42:45.376121+0000","last_active":"2024-04-16T07:42:58.329672+0000","last_peered":"2024-04-16T07:42:58.329672+0000","last_clean":"2024-04-16T07:42:58.329672+0000","last_became_active":"2024-04-16T07:42:45.370390+0000","last_became_peered":"2024-04-16T07:42:45.370390+0000","last_unstale":"2024-04-16T07:42:58.329672+0000","last_undegraded":"2024-04-16T07:42:58.329672+0000","last_fullsized":"2024-04-16T07:42:58.329672+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":164429824,"num_objects":2509,"num_object_clones":0,"num_object_copies":7527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2509,"num_write_kb":160576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2493,"ondisk_log_size":2493,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":43376,"kb_used_data":8232,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94328464,"statfs":{"total":96636764160,"available":96592347136,"internally_reserved":0,"allocated":8429568,"data_stored":132227877,"data_compressed":6277871,"data_compressed_allocated":8265728,"data_compressed_original":132169728,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37424,"kb_used_data":6440,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94334416,"statfs":{"total":96636764160,"available":96598441984,"internally_reserved":0,"allocated":6594560,"data_stored":102867749,"data_compressed":4883928,"data_compressed_allocated":6430720,"data_compressed_original":102809600,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38068,"kb_used_data":7084,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94333772,"statfs":{"total":96636764160,"available":96597782528,"internally_reserved":0,"allocated":7254016,"data_stored":113811717,"data_compressed":5401578,"data_compressed_allocated":7110656,"data_compressed_original":113770496,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37564,"kb_used_data":6580,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94334276,"statfs":{"total":96636764160,"available":96598298624,"internally_reserved":0,"allocated":6737920,"data_stored":105161509,"data_compressed":4992837,"data_compressed_allocated":6574080,"data_compressed_original":105103360,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:43:12.393 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 14 16 2952 2936 13.1059 13.0625 0.121058 0.0759877 2024-04-16T07:43:12.785 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:43:12.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:12.931+0000 7f34736ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:12.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:12.940+0000 7f34736ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:13.393 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 15 16 3160 3144 13.0988 13 0.0707909 0.0760957 2024-04-16T07:43:14.393 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 16 16 3357 3341 13.0496 12.3125 0.0994734 0.0763602 2024-04-16T07:43:15.301 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:43:15.393 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 17 16 3560 3544 13.0282 12.6875 0.0646455 0.0765427 2024-04-16T07:43:15.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:15.443+0000 7f841a013640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:15.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:15.454+0000 7f841a013640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:16.393 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 18 16 3760 3744 12.9988 12.5 0.0860256 0.0767423 2024-04-16T07:43:17.370 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-16T07:43:17.370 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:43:17.370 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 2, 3] 2024-04-16T07:43:17.371 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-16T07:43:17.371 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:43:17.393 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 19 16 3964 3948 12.9857 12.75 0.0300818 0.0767598 2024-04-16T07:43:17.447 INFO:tasks.ceph.osd.3:Stopped 2024-04-16T07:43:17.860 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:43:17.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:17.981+0000 7effd1bfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:17.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:17.987+0000 7effd1bfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:18.394 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:2024-04-16T07:43:18.395480+0000 min lat: 0.0168854 max lat: 0.178527 avg lat: 0.0767598 2024-04-16T07:43:18.394 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:43:18.394 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 20 16 3964 3948 12.3364 0 - 0.0767598 2024-04-16T07:43:19.394 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 21 16 3964 3948 11.7489 0 - 0.0767598 2024-04-16T07:43:20.394 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 22 16 3964 3948 11.2148 0 - 0.0767598 2024-04-16T07:43:21.394 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 23 16 3964 3948 10.7272 0 - 0.0767598 2024-04-16T07:43:22.394 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 24 16 3964 3948 10.2802 0 - 0.0767598 2024-04-16T07:43:22.448 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2] 2024-04-16T07:43:22.448 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:43:22.448 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:43:22.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:22.574+0000 7f28bab5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:22.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:22.583+0000 7f28bab5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:22.784 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:43:22.784 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-16T07:43:22.796 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-04-16T07:43:21.224237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":251920928,"num_objects":3837,"num_object_clones":0,"num_object_copies":11511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3972,"num_write_kb":247334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3916,"ondisk_log_size":3916,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":194948,"kb_used_data":46052,"kb_used_omap":0,"kb_used_meta":148864,"kb_avail":377292412,"statfs":{"total":386547056640,"available":386347429888,"internally_reserved":0,"allocated":47157248,"data_stored":744327796,"data_compressed":35339262,"data_compressed_allocated":46522368,"data_compressed_original":744112128,"omap_allocated":0,"internal_metadata":152436736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":155254784,"num_objects":2369,"num_object_clones":0,"num_object_copies":7107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2369,"num_write_kb":151616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002684"},"pg_stats":[{"pgid":"3.f","version":"17'232","reported_seq":245,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:17.177967+0000","last_change":"2024-04-16T07:42:57.456939+0000","last_active":"2024-04-16T07:43:17.177967+0000","last_peered":"2024-04-16T07:43:17.177967+0000","last_clean":"2024-04-16T07:43:17.177967+0000","last_became_active":"2024-04-16T07:42:57.450056+0000","last_became_peered":"2024-04-16T07:42:57.450056+0000","last_unstale":"2024-04-16T07:43:17.177967+0000","last_undegraded":"2024-04-16T07:43:17.177967+0000","last_fullsized":"2024-04-16T07:43:17.177967+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15269888,"num_objects":233,"num_object_clones":0,"num_object_copies":699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":233,"num_write_kb":14912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"17'236","reported_seq":249,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:17.354112+0000","last_change":"2024-04-16T07:42:57.453661+0000","last_active":"2024-04-16T07:43:17.354112+0000","last_peered":"2024-04-16T07:43:17.354112+0000","last_clean":"2024-04-16T07:43:17.354112+0000","last_became_active":"2024-04-16T07:42:57.447818+0000","last_became_peered":"2024-04-16T07:42:57.447818+0000","last_unstale":"2024-04-16T07:43:17.354112+0000","last_undegraded":"2024-04-16T07:43:17.354112+0000","last_fullsized":"2024-04-16T07:43:17.354112+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15532032,"num_objects":237,"num_object_clones":0,"num_object_copies":711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":237,"num_write_kb":15168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'241","reported_seq":254,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:17.035704+0000","last_change":"2024-04-16T07:42:57.507834+0000","last_active":"2024-04-16T07:43:17.035704+0000","last_peered":"2024-04-16T07:43:17.035704+0000","last_clean":"2024-04-16T07:43:17.035704+0000","last_became_active":"2024-04-16T07:42:57.499895+0000","last_became_peered":"2024-04-16T07:42:57.499895+0000","last_unstale":"2024-04-16T07:43:17.035704+0000","last_undegraded":"2024-04-16T07:43:17.035704+0000","last_fullsized":"2024-04-16T07:43:17.035704+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15859712,"num_objects":242,"num_object_clones":0,"num_object_copies":726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":242,"num_write_kb":15488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'257","reported_seq":270,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:17.143826+0000","last_change":"2024-04-16T07:42:57.465377+0000","last_active":"2024-04-16T07:43:17.143826+0000","last_peered":"2024-04-16T07:43:17.143826+0000","last_clean":"2024-04-16T07:43:17.143826+0000","last_became_active":"2024-04-16T07:42:57.451654+0000","last_became_peered":"2024-04-16T07:42:57.451654+0000","last_unstale":"2024-04-16T07:43:17.143826+0000","last_undegraded":"2024-04-16T07:43:17.143826+0000","last_fullsized":"2024-04-16T07:43:17.143826+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16908288,"num_objects":258,"num_object_clones":0,"num_object_copies":774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":258,"num_write_kb":16512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'213","reported_seq":226,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:15.068754+0000","last_change":"2024-04-16T07:42:57.459819+0000","last_active":"2024-04-16T07:43:15.068754+0000","last_peered":"2024-04-16T07:43:15.068754+0000","last_clean":"2024-04-16T07:43:15.068754+0000","last_became_active":"2024-04-16T07:42:57.453218+0000","last_became_peered":"2024-04-16T07:42:57.453218+0000","last_unstale":"2024-04-16T07:43:15.068754+0000","last_undegraded":"2024-04-16T07:43:15.068754+0000","last_fullsized":"2024-04-16T07:43:15.068754+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14024704,"num_objects":214,"num_object_clones":0,"num_object_copies":642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":214,"num_write_kb":13696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'262","reported_seq":275,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:17.299731+0000","last_change":"2024-04-16T07:42:57.489933+0000","last_active":"2024-04-16T07:43:17.299731+0000","last_peered":"2024-04-16T07:43:17.299731+0000","last_clean":"2024-04-16T07:43:17.299731+0000","last_became_active":"2024-04-16T07:42:57.482517+0000","last_became_peered":"2024-04-16T07:42:57.482517+0000","last_unstale":"2024-04-16T07:43:17.299731+0000","last_undegraded":"2024-04-16T07:43:17.299731+0000","last_fullsized":"2024-04-16T07:43:17.299731+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17235968,"num_objects":263,"num_object_clones":0,"num_object_copies":789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":263,"num_write_kb":16832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'233","reported_seq":246,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:17.218620+0000","last_change":"2024-04-16T07:42:57.469015+0000","last_active":"2024-04-16T07:43:17.218620+0000","last_peered":"2024-04-16T07:43:17.218620+0000","last_clean":"2024-04-16T07:43:17.218620+0000","last_became_active":"2024-04-16T07:42:57.460537+0000","last_became_peered":"2024-04-16T07:42:57.460537+0000","last_unstale":"2024-04-16T07:43:17.218620+0000","last_undegraded":"2024-04-16T07:43:17.218620+0000","last_fullsized":"2024-04-16T07:43:17.218620+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15335424,"num_objects":234,"num_object_clones":0,"num_object_copies":702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":234,"num_write_kb":14976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"17'242","reported_seq":255,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:17.293834+0000","last_change":"2024-04-16T07:42:57.459433+0000","last_active":"2024-04-16T07:43:17.293834+0000","last_peered":"2024-04-16T07:43:17.293834+0000","last_clean":"2024-04-16T07:43:17.293834+0000","last_became_active":"2024-04-16T07:42:57.452612+0000","last_became_peered":"2024-04-16T07:42:57.452612+0000","last_unstale":"2024-04-16T07:43:17.293834+0000","last_undegraded":"2024-04-16T07:43:17.293834+0000","last_fullsized":"2024-04-16T07:43:17.293834+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15925248,"num_objects":243,"num_object_clones":0,"num_object_copies":729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":243,"num_write_kb":15552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":154,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.415419+0000","last_change":"2024-04-16T07:42:43.287780+0000","last_active":"2024-04-16T07:42:58.415419+0000","last_peered":"2024-04-16T07:42:58.415419+0000","last_clean":"2024-04-16T07:42:58.415419+0000","last_became_active":"2024-04-16T07:42:43.282375+0000","last_became_peered":"2024-04-16T07:42:43.282375+0000","last_unstale":"2024-04-16T07:42:58.415419+0000","last_undegraded":"2024-04-16T07:42:58.415419+0000","last_fullsized":"2024-04-16T07:42:58.415419+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_clean_scrub_stamp":"2024-04-16T07:42:42.229959+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.331433+0000","last_change":"2024-04-16T07:42:45.350220+0000","last_active":"2024-04-16T07:42:58.331433+0000","last_peered":"2024-04-16T07:42:58.331433+0000","last_clean":"2024-04-16T07:42:58.331433+0000","last_became_active":"2024-04-16T07:42:45.343569+0000","last_became_peered":"2024-04-16T07:42:45.343569+0000","last_unstale":"2024-04-16T07:42:58.331433+0000","last_undegraded":"2024-04-16T07:42:58.331433+0000","last_fullsized":"2024-04-16T07:42:58.331433+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"17'254","reported_seq":267,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:17.332789+0000","last_change":"2024-04-16T07:42:57.476764+0000","last_active":"2024-04-16T07:43:17.332789+0000","last_peered":"2024-04-16T07:43:17.332789+0000","last_clean":"2024-04-16T07:43:17.332789+0000","last_became_active":"2024-04-16T07:42:57.471552+0000","last_became_peered":"2024-04-16T07:42:57.471552+0000","last_unstale":"2024-04-16T07:43:17.332789+0000","last_undegraded":"2024-04-16T07:43:17.332789+0000","last_fullsized":"2024-04-16T07:43:17.332789+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16711680,"num_objects":255,"num_object_clones":0,"num_object_copies":765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":255,"num_write_kb":16320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.426126+0000","last_change":"2024-04-16T07:42:45.334636+0000","last_active":"2024-04-16T07:42:58.426126+0000","last_peered":"2024-04-16T07:42:58.426126+0000","last_clean":"2024-04-16T07:42:58.426126+0000","last_became_active":"2024-04-16T07:42:45.329643+0000","last_became_peered":"2024-04-16T07:42:45.329643+0000","last_unstale":"2024-04-16T07:42:58.426126+0000","last_undegraded":"2024-04-16T07:42:58.426126+0000","last_fullsized":"2024-04-16T07:42:58.426126+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"17'241","reported_seq":254,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:15.202074+0000","last_change":"2024-04-16T07:42:57.533906+0000","last_active":"2024-04-16T07:43:15.202074+0000","last_peered":"2024-04-16T07:43:15.202074+0000","last_clean":"2024-04-16T07:43:15.202074+0000","last_became_active":"2024-04-16T07:42:57.527972+0000","last_became_peered":"2024-04-16T07:42:57.527972+0000","last_unstale":"2024-04-16T07:43:15.202074+0000","last_undegraded":"2024-04-16T07:43:15.202074+0000","last_fullsized":"2024-04-16T07:43:15.202074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15859712,"num_objects":242,"num_object_clones":0,"num_object_copies":726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":242,"num_write_kb":15488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.317746+0000","last_change":"2024-04-16T07:42:45.327681+0000","last_active":"2024-04-16T07:42:58.317746+0000","last_peered":"2024-04-16T07:42:58.317746+0000","last_clean":"2024-04-16T07:42:58.317746+0000","last_became_active":"2024-04-16T07:42:45.318385+0000","last_became_peered":"2024-04-16T07:42:45.318385+0000","last_unstale":"2024-04-16T07:42:58.317746+0000","last_undegraded":"2024-04-16T07:42:58.317746+0000","last_fullsized":"2024-04-16T07:42:58.317746+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'232","reported_seq":245,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:15.143777+0000","last_change":"2024-04-16T07:42:57.487393+0000","last_active":"2024-04-16T07:43:15.143777+0000","last_peered":"2024-04-16T07:43:15.143777+0000","last_clean":"2024-04-16T07:43:15.143777+0000","last_became_active":"2024-04-16T07:42:57.481110+0000","last_became_peered":"2024-04-16T07:42:57.481110+0000","last_unstale":"2024-04-16T07:43:15.143777+0000","last_undegraded":"2024-04-16T07:43:15.143777+0000","last_fullsized":"2024-04-16T07:43:15.143777+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15269888,"num_objects":233,"num_object_clones":0,"num_object_copies":699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":233,"num_write_kb":14912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.419174+0000","last_change":"2024-04-16T07:42:45.357843+0000","last_active":"2024-04-16T07:42:58.419174+0000","last_peered":"2024-04-16T07:42:58.419174+0000","last_clean":"2024-04-16T07:42:58.419174+0000","last_became_active":"2024-04-16T07:42:45.348908+0000","last_became_peered":"2024-04-16T07:42:45.348908+0000","last_unstale":"2024-04-16T07:42:58.419174+0000","last_undegraded":"2024-04-16T07:42:58.419174+0000","last_fullsized":"2024-04-16T07:42:58.419174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"17'227","reported_seq":240,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:17.285755+0000","last_change":"2024-04-16T07:42:57.437012+0000","last_active":"2024-04-16T07:43:17.285755+0000","last_peered":"2024-04-16T07:43:17.285755+0000","last_clean":"2024-04-16T07:43:17.285755+0000","last_became_active":"2024-04-16T07:42:57.429704+0000","last_became_peered":"2024-04-16T07:42:57.429704+0000","last_unstale":"2024-04-16T07:43:17.285755+0000","last_undegraded":"2024-04-16T07:43:17.285755+0000","last_fullsized":"2024-04-16T07:43:17.285755+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14942208,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":228,"num_write_kb":14592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'239","reported_seq":252,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:17.223664+0000","last_change":"2024-04-16T07:42:57.463631+0000","last_active":"2024-04-16T07:43:17.223664+0000","last_peered":"2024-04-16T07:43:17.223664+0000","last_clean":"2024-04-16T07:43:17.223664+0000","last_became_active":"2024-04-16T07:42:57.453387+0000","last_became_peered":"2024-04-16T07:42:57.453387+0000","last_unstale":"2024-04-16T07:43:17.223664+0000","last_undegraded":"2024-04-16T07:43:17.223664+0000","last_fullsized":"2024-04-16T07:43:17.223664+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":239,"log_dups_size":0,"ondisk_log_size":239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728640,"num_objects":240,"num_object_clones":0,"num_object_copies":720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":240,"num_write_kb":15360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.434405+0000","last_change":"2024-04-16T07:42:45.340605+0000","last_active":"2024-04-16T07:42:58.434405+0000","last_peered":"2024-04-16T07:42:58.434405+0000","last_clean":"2024-04-16T07:42:58.434405+0000","last_became_active":"2024-04-16T07:42:45.335238+0000","last_became_peered":"2024-04-16T07:42:45.335238+0000","last_unstale":"2024-04-16T07:42:58.434405+0000","last_undegraded":"2024-04-16T07:42:58.434405+0000","last_fullsized":"2024-04-16T07:42:58.434405+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"17'234","reported_seq":247,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:17.310625+0000","last_change":"2024-04-16T07:42:57.470835+0000","last_active":"2024-04-16T07:43:17.310625+0000","last_peered":"2024-04-16T07:43:17.310625+0000","last_clean":"2024-04-16T07:43:17.310625+0000","last_became_active":"2024-04-16T07:42:57.460356+0000","last_became_peered":"2024-04-16T07:42:57.460356+0000","last_unstale":"2024-04-16T07:43:17.310625+0000","last_undegraded":"2024-04-16T07:43:17.310625+0000","last_fullsized":"2024-04-16T07:43:17.310625+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":234,"log_dups_size":0,"ondisk_log_size":234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15400960,"num_objects":235,"num_object_clones":0,"num_object_copies":705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":235,"num_write_kb":15040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.430163+0000","last_change":"2024-04-16T07:42:45.332152+0000","last_active":"2024-04-16T07:42:58.430163+0000","last_peered":"2024-04-16T07:42:58.430163+0000","last_clean":"2024-04-16T07:42:58.430163+0000","last_became_active":"2024-04-16T07:42:45.325833+0000","last_became_peered":"2024-04-16T07:42:45.325833+0000","last_unstale":"2024-04-16T07:42:58.430163+0000","last_undegraded":"2024-04-16T07:42:58.430163+0000","last_fullsized":"2024-04-16T07:42:58.430163+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'234","reported_seq":247,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:14.958350+0000","last_change":"2024-04-16T07:42:57.489443+0000","last_active":"2024-04-16T07:43:14.958350+0000","last_peered":"2024-04-16T07:43:14.958350+0000","last_clean":"2024-04-16T07:43:14.958350+0000","last_became_active":"2024-04-16T07:42:57.480103+0000","last_became_peered":"2024-04-16T07:42:57.480103+0000","last_unstale":"2024-04-16T07:43:14.958350+0000","last_undegraded":"2024-04-16T07:43:14.958350+0000","last_fullsized":"2024-04-16T07:43:14.958350+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":234,"log_dups_size":0,"ondisk_log_size":234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15400960,"num_objects":235,"num_object_clones":0,"num_object_copies":705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":235,"num_write_kb":15040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.410936+0000","last_change":"2024-04-16T07:42:45.333004+0000","last_active":"2024-04-16T07:42:58.410936+0000","last_peered":"2024-04-16T07:42:58.410936+0000","last_clean":"2024-04-16T07:42:58.410936+0000","last_became_active":"2024-04-16T07:42:45.323737+0000","last_became_peered":"2024-04-16T07:42:45.323737+0000","last_unstale":"2024-04-16T07:42:58.410936+0000","last_undegraded":"2024-04-16T07:42:58.410936+0000","last_fullsized":"2024-04-16T07:42:58.410936+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'242","reported_seq":255,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:43:17.210150+0000","last_change":"2024-04-16T07:42:57.475347+0000","last_active":"2024-04-16T07:43:17.210150+0000","last_peered":"2024-04-16T07:43:17.210150+0000","last_clean":"2024-04-16T07:43:17.210150+0000","last_became_active":"2024-04-16T07:42:57.465803+0000","last_became_peered":"2024-04-16T07:42:57.465803+0000","last_unstale":"2024-04-16T07:43:17.210150+0000","last_undegraded":"2024-04-16T07:43:17.210150+0000","last_fullsized":"2024-04-16T07:43:17.210150+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15925248,"num_objects":243,"num_object_clones":0,"num_object_copies":729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":243,"num_write_kb":15552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-16T07:42:58.329672+0000","last_change":"2024-04-16T07:42:45.376121+0000","last_active":"2024-04-16T07:42:58.329672+0000","last_peered":"2024-04-16T07:42:58.329672+0000","last_clean":"2024-04-16T07:42:58.329672+0000","last_became_active":"2024-04-16T07:42:45.370390+0000","last_became_peered":"2024-04-16T07:42:45.370390+0000","last_unstale":"2024-04-16T07:42:58.329672+0000","last_undegraded":"2024-04-16T07:42:58.329672+0000","last_fullsized":"2024-04-16T07:42:58.329672+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":251330560,"num_objects":3835,"num_object_clones":0,"num_object_copies":11505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3835,"num_write_kb":245440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3819,"ondisk_log_size":3819,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51148,"kb_used_data":11844,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94320692,"statfs":{"total":96636764160,"available":96584388608,"internally_reserved":0,"allocated":12128256,"data_stored":191406885,"data_compressed":9088007,"data_compressed_allocated":11964416,"data_compressed_original":191348736,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":46116,"kb_used_data":10972,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94325724,"statfs":{"total":96636764160,"available":96589541376,"internally_reserved":0,"allocated":11235328,"data_stored":177120037,"data_compressed":8409824,"data_compressed_allocated":11071488,"data_compressed_original":177061888,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51432,"kb_used_data":12128,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94320408,"statfs":{"total":96636764160,"available":96584097792,"internally_reserved":0,"allocated":12419072,"data_stored":196452613,"data_compressed":9325810,"data_compressed_allocated":12275712,"data_compressed_original":196411392,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":46252,"kb_used_data":11108,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94325588,"statfs":{"total":96636764160,"available":96589402112,"internally_reserved":0,"allocated":11374592,"data_stored":179348261,"data_compressed":8515621,"data_compressed_allocated":11210752,"data_compressed_original":179290112,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:43:23.395 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 25 16 3964 3948 9.86897 0 - 0.0767598 2024-04-16T07:43:24.394 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 26 16 3964 3948 9.48937 0 - 0.0767598 2024-04-16T07:43:25.395 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 27 16 3964 3948 9.1379 0 - 0.0767598 2024-04-16T07:43:26.395 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 28 16 3964 3948 8.81153 0 - 0.0767598 2024-04-16T07:43:27.395 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 29 16 3964 3948 8.50766 0 - 0.0767598 2024-04-16T07:43:27.798 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2] 2024-04-16T07:43:27.799 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:43:27.799 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-04-16T07:43:27.799 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-16T07:43:27.799 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-16T07:43:27.802 INFO:tasks.ceph.osd.3:Started 2024-04-16T07:43:27.802 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:43:27.916 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:43:27.919 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:43:27.920 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:43:28.395 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 30 16 3964 3948 8.22407 0 - 0.0767598 2024-04-16T07:43:29.395 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 31 16 3964 3948 7.95877 0 - 0.0767598 2024-04-16T07:43:30.395 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 32 16 3964 3948 7.71006 0 - 0.0767598 2024-04-16T07:43:31.396 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 33 16 3964 3948 7.47641 0 - 0.0767598 2024-04-16T07:43:32.396 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 34 16 3964 3948 7.2565 0 - 0.0767598 2024-04-16T07:43:32.921 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:43:33.026 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:43:33.029 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:43:33.029 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:43:33.396 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 35 16 3964 3948 7.04917 0 - 0.0767598 2024-04-16T07:43:34.396 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 36 16 3964 3948 6.85336 0 - 0.0767598 2024-04-16T07:43:35.396 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 37 16 3964 3948 6.66813 0 - 0.0767598 2024-04-16T07:43:36.396 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 38 16 3964 3948 6.49264 0 - 0.0767598 2024-04-16T07:43:37.397 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 39 16 3964 3948 6.32616 0 - 0.0767598 2024-04-16T07:43:38.030 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:43:38.134 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:43:38.137 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:43:38.138 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:43:38.397 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:2024-04-16T07:43:38.398447+0000 min lat: 0.0168854 max lat: 0.178527 avg lat: 0.0767598 2024-04-16T07:43:38.397 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:43:38.397 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 40 16 3964 3948 6.168 0 - 0.0767598 2024-04-16T07:43:39.397 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 41 16 3964 3948 6.01756 0 - 0.0767598 2024-04-16T07:43:39.939 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:43:39.945 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:43:39.945 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:43:40.397 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 42 16 3964 3948 5.87428 0 - 0.0767598 2024-04-16T07:43:41.397 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 43 16 3964 3948 5.73766 0 - 0.0767598 2024-04-16T07:43:41.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:43:42.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:42.116+0000 7fce68d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:42.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:42.129+0000 7fce68d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:42.397 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 44 16 3985 3969 5.63709 0.0525 25.0387 0.151998 2024-04-16T07:43:43.138 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:43:43.321 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-16T07:43:43.321 INFO:teuthology.orchestra.run.smithi094.stdout: "ops_in_flight": [ 2024-04-16T07:43:43.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.322 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.322 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777328, 2024-04-16T07:43:43.322 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.322 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.322 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.237987+0000" 2024-04-16T07:43:43.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.322 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.322 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.322 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.238019+0000" 2024-04-16T07:43:43.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.322 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.322 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.323 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.238236+0000" 2024-04-16T07:43:43.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.323 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.323 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.323 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.323 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.323 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.323 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.238453+0000" 2024-04-16T07:43:43.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.323 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.323 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.323 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.240057+0000" 2024-04-16T07:43:43.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.323 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.324 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.324 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.324 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.324 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.324 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.324 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.324 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.324 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777330, 2024-04-16T07:43:43.324 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.324 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.325 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.281037+0000" 2024-04-16T07:43:43.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.325 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.325 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.325 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.281059+0000" 2024-04-16T07:43:43.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.325 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.325 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.325 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.281277+0000" 2024-04-16T07:43:43.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.325 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.325 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.325 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.325 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.325 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.326 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.281440+0000" 2024-04-16T07:43:43.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.326 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.326 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.326 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.286610+0000" 2024-04-16T07:43:43.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.326 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.326 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.326 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.326 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.326 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.326 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.326 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.327 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "replicated_request", 2024-04-16T07:43:43.327 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 319, 2024-04-16T07:43:43.327 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.327 INFO:teuthology.orchestra.run.smithi094.stdout: "RepRequest": { 2024-04-16T07:43:43.327 INFO:teuthology.orchestra.run.smithi094.stdout: "reqid": "client.4276.0:4152", 2024-04-16T07:43:43.327 INFO:teuthology.orchestra.run.smithi094.stdout: "pgid": "3.4", 2024-04-16T07:43:43.327 INFO:teuthology.orchestra.run.smithi094.stdout: "map_epoch": 20, 2024-04-16T07:43:43.327 INFO:teuthology.orchestra.run.smithi094.stdout: "min_epoch": 19, 2024-04-16T07:43:43.327 INFO:teuthology.orchestra.run.smithi094.stdout: "oid": "3:2d6c8545:::benchmark_data_smithi031_32265_object4151:head", 2024-04-16T07:43:43.327 INFO:teuthology.orchestra.run.smithi094.stdout: "from": "2" 2024-04-16T07:43:43.327 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.327 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.327 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "replicated_request", 2024-04-16T07:43:43.328 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 320, 2024-04-16T07:43:43.328 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.328 INFO:teuthology.orchestra.run.smithi094.stdout: "RepRequest": { 2024-04-16T07:43:43.328 INFO:teuthology.orchestra.run.smithi094.stdout: "reqid": "client.4276.0:4141", 2024-04-16T07:43:43.328 INFO:teuthology.orchestra.run.smithi094.stdout: "pgid": "3.2", 2024-04-16T07:43:43.328 INFO:teuthology.orchestra.run.smithi094.stdout: "map_epoch": 20, 2024-04-16T07:43:43.328 INFO:teuthology.orchestra.run.smithi094.stdout: "min_epoch": 19, 2024-04-16T07:43:43.328 INFO:teuthology.orchestra.run.smithi094.stdout: "oid": "3:48e077f5:::benchmark_data_smithi031_32265_object4140:head", 2024-04-16T07:43:43.328 INFO:teuthology.orchestra.run.smithi094.stdout: "from": "1" 2024-04-16T07:43:43.328 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.328 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.328 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.328 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777268, 2024-04-16T07:43:43.328 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.329 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.329 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.360153+0000" 2024-04-16T07:43:43.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.329 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.329 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.329 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.360194+0000" 2024-04-16T07:43:43.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.329 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.329 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.329 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.360435+0000" 2024-04-16T07:43:43.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.329 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.329 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.329 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.330 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.330 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.330 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.360626+0000" 2024-04-16T07:43:43.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.330 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.330 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.330 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.361137+0000" 2024-04-16T07:43:43.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.330 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.330 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.330 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.331 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.331 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.363910+0000" 2024-04-16T07:43:43.331 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.331 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.331 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.331 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777270, 2024-04-16T07:43:43.331 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.331 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.331 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.362655+0000" 2024-04-16T07:43:43.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.332 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.332 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.332 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.362666+0000" 2024-04-16T07:43:43.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.332 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.332 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.332 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.362863+0000" 2024-04-16T07:43:43.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.332 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.332 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.332 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.332 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.332 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.333 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.363014+0000" 2024-04-16T07:43:43.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.333 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.333 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.333 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.364214+0000" 2024-04-16T07:43:43.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.333 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.333 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.333 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.333 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.333 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.366914+0000" 2024-04-16T07:43:43.333 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.333 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.334 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.334 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777275, 2024-04-16T07:43:43.334 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.334 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.334 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.366672+0000" 2024-04-16T07:43:43.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.334 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.334 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.335 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.366679+0000" 2024-04-16T07:43:43.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.335 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.335 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.335 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.366852+0000" 2024-04-16T07:43:43.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.335 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.335 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.335 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.335 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.335 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.335 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.366990+0000" 2024-04-16T07:43:43.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.335 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.336 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.336 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.368325+0000" 2024-04-16T07:43:43.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.336 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.336 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.336 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.336 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.336 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.471333+0000" 2024-04-16T07:43:43.336 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.336 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.337 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.337 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777274, 2024-04-16T07:43:43.337 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.337 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.337 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.366078+0000" 2024-04-16T07:43:43.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.337 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.337 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.337 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.366086+0000" 2024-04-16T07:43:43.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.337 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.337 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.337 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.366225+0000" 2024-04-16T07:43:43.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.338 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.338 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.338 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.338 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.338 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.338 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.366388+0000" 2024-04-16T07:43:43.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.338 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.338 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.338 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.367220+0000" 2024-04-16T07:43:43.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.339 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.339 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.339 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.339 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.339 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.487011+0000" 2024-04-16T07:43:43.339 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.339 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.339 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.339 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777279, 2024-04-16T07:43:43.339 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.340 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.340 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.369853+0000" 2024-04-16T07:43:43.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.340 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.340 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.340 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.369860+0000" 2024-04-16T07:43:43.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.340 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.340 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.340 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.370000+0000" 2024-04-16T07:43:43.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.340 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.340 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.341 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.341 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.341 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.341 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.370160+0000" 2024-04-16T07:43:43.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.341 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.341 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.341 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.370986+0000" 2024-04-16T07:43:43.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.341 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.341 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.341 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.342 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.342 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.493878+0000" 2024-04-16T07:43:43.342 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.342 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.342 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.342 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777281, 2024-04-16T07:43:43.342 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.342 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.342 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.371255+0000" 2024-04-16T07:43:43.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.343 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.343 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.343 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.371266+0000" 2024-04-16T07:43:43.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.343 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.343 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.343 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.371510+0000" 2024-04-16T07:43:43.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.343 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.343 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.343 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.343 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.343 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.344 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.371638+0000" 2024-04-16T07:43:43.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.344 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.344 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.344 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.378154+0000" 2024-04-16T07:43:43.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.344 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.344 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.344 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.344 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.344 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.497001+0000" 2024-04-16T07:43:43.344 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.344 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.345 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.345 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777282, 2024-04-16T07:43:43.345 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.345 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.345 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.372309+0000" 2024-04-16T07:43:43.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.345 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.345 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.345 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.372321+0000" 2024-04-16T07:43:43.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.345 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.345 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.345 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.372576+0000" 2024-04-16T07:43:43.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.346 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.346 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.346 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.346 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.346 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.346 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.372703+0000" 2024-04-16T07:43:43.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.346 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.346 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.346 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.380312+0000" 2024-04-16T07:43:43.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.346 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.347 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.347 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.347 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.347 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.499117+0000" 2024-04-16T07:43:43.347 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.347 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.347 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.347 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777280, 2024-04-16T07:43:43.347 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.348 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.348 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.370626+0000" 2024-04-16T07:43:43.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.348 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.348 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.348 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.370633+0000" 2024-04-16T07:43:43.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.348 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.348 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.348 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.370789+0000" 2024-04-16T07:43:43.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.348 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.348 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.348 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.349 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.349 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.349 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.370885+0000" 2024-04-16T07:43:43.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.349 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.349 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.349 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.377278+0000" 2024-04-16T07:43:43.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.349 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.349 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.349 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.349 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.349 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.501799+0000" 2024-04-16T07:43:43.350 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.350 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.350 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.350 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777283, 2024-04-16T07:43:43.350 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.350 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.350 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.374240+0000" 2024-04-16T07:43:43.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.350 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.350 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.350 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.374254+0000" 2024-04-16T07:43:43.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.351 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.351 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.351 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.374476+0000" 2024-04-16T07:43:43.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.351 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.351 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.351 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.351 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.351 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.351 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.374610+0000" 2024-04-16T07:43:43.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.351 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.351 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.352 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.381200+0000" 2024-04-16T07:43:43.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.353 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.353 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.353 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.353 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.354 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.513058+0000" 2024-04-16T07:43:43.354 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.354 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.354 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.354 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777284, 2024-04-16T07:43:43.354 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.354 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.354 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.432425+0000" 2024-04-16T07:43:43.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.354 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.355 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.355 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.432436+0000" 2024-04-16T07:43:43.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.355 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.355 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.355 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.432585+0000" 2024-04-16T07:43:43.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.355 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.355 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.355 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.355 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.355 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.355 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.432676+0000" 2024-04-16T07:43:43.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.356 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.356 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.356 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.436350+0000" 2024-04-16T07:43:43.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.356 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.356 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.356 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.356 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.356 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.523732+0000" 2024-04-16T07:43:43.356 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.356 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.357 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.357 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777288, 2024-04-16T07:43:43.357 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.357 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.357 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.500858+0000" 2024-04-16T07:43:43.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.357 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.357 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.357 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.500869+0000" 2024-04-16T07:43:43.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.357 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.357 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.357 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.501141+0000" 2024-04-16T07:43:43.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.358 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.358 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.358 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.358 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.358 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.358 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.501315+0000" 2024-04-16T07:43:43.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.358 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.358 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.358 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.503539+0000" 2024-04-16T07:43:43.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.358 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.359 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.359 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.359 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.359 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.586064+0000" 2024-04-16T07:43:43.359 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.359 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.359 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.359 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777292, 2024-04-16T07:43:43.359 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.359 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.360 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.562797+0000" 2024-04-16T07:43:43.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.360 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.360 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.360 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.562812+0000" 2024-04-16T07:43:43.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.360 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.360 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.360 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.563028+0000" 2024-04-16T07:43:43.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.360 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.360 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.361 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.361 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.361 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.361 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.563215+0000" 2024-04-16T07:43:43.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.361 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.361 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.361 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.565431+0000" 2024-04-16T07:43:43.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.361 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.361 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.361 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.678105+0000" 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777293, 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.590823+0000" 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.590837+0000" 2024-04-16T07:43:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.590987+0000" 2024-04-16T07:43:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.591136+0000" 2024-04-16T07:43:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.592826+0000" 2024-04-16T07:43:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.699381+0000" 2024-04-16T07:43:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777294, 2024-04-16T07:43:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.608356+0000" 2024-04-16T07:43:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.608373+0000" 2024-04-16T07:43:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.608597+0000" 2024-04-16T07:43:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.366 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.366 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.366 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.366 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.366 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.608733+0000" 2024-04-16T07:43:43.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.366 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.366 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.366 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.610513+0000" 2024-04-16T07:43:43.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.366 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.366 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.366 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.367 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.367 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.708400+0000" 2024-04-16T07:43:43.367 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.367 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.367 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.367 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777296, 2024-04-16T07:43:43.367 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.367 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.367 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.625676+0000" 2024-04-16T07:43:43.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.367 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.368 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.368 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.625693+0000" 2024-04-16T07:43:43.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.368 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.368 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.368 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.625918+0000" 2024-04-16T07:43:43.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.368 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.368 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.368 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.368 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.368 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.368 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.626104+0000" 2024-04-16T07:43:43.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.369 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.369 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.369 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.628813+0000" 2024-04-16T07:43:43.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.369 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.369 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.369 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.369 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.369 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.714020+0000" 2024-04-16T07:43:43.369 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.369 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.369 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.370 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777297, 2024-04-16T07:43:43.370 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.370 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.370 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.654470+0000" 2024-04-16T07:43:43.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.370 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.370 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.370 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.654488+0000" 2024-04-16T07:43:43.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.370 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.370 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.370 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.654668+0000" 2024-04-16T07:43:43.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.371 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.371 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.371 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.371 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.371 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.371 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.654844+0000" 2024-04-16T07:43:43.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.371 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.371 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.371 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.656109+0000" 2024-04-16T07:43:43.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.371 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.371 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.372 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.372 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.372 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.746882+0000" 2024-04-16T07:43:43.372 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.372 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.372 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.372 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777298, 2024-04-16T07:43:43.372 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.372 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.373 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.712620+0000" 2024-04-16T07:43:43.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.373 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.373 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.373 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.712635+0000" 2024-04-16T07:43:43.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.373 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.373 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.373 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.712851+0000" 2024-04-16T07:43:43.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.373 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.373 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.373 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.373 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.374 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.374 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.712997+0000" 2024-04-16T07:43:43.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.374 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.374 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.374 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.716686+0000" 2024-04-16T07:43:43.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.374 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.374 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.374 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.374 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.374 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.827052+0000" 2024-04-16T07:43:43.375 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.375 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.375 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.375 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777300, 2024-04-16T07:43:43.375 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.375 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.375 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.721685+0000" 2024-04-16T07:43:43.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.375 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.375 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.375 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.721709+0000" 2024-04-16T07:43:43.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.375 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.375 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.376 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.724273+0000" 2024-04-16T07:43:43.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.376 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.376 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.376 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.376 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.376 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.376 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.724464+0000" 2024-04-16T07:43:43.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.376 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.376 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.376 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.727407+0000" 2024-04-16T07:43:43.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.376 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.377 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.377 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.377 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.377 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.836620+0000" 2024-04-16T07:43:43.377 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.377 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.377 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.377 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777299, 2024-04-16T07:43:43.377 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.377 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.377 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.717612+0000" 2024-04-16T07:43:43.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.378 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.378 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.378 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.717626+0000" 2024-04-16T07:43:43.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.378 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.378 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.378 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.717850+0000" 2024-04-16T07:43:43.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.378 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.378 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.378 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.378 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.378 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.378 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.718002+0000" 2024-04-16T07:43:43.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.379 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.379 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.379 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.722925+0000" 2024-04-16T07:43:43.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.379 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.379 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.379 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.379 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.379 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.843592+0000" 2024-04-16T07:43:43.379 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.379 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.379 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.380 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777303, 2024-04-16T07:43:43.380 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.380 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.380 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.792559+0000" 2024-04-16T07:43:43.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.380 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.380 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.380 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.792587+0000" 2024-04-16T07:43:43.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.380 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.380 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.380 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.792861+0000" 2024-04-16T07:43:43.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.380 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.381 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.381 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.381 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.381 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.381 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.793014+0000" 2024-04-16T07:43:43.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.381 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.381 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.381 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.796903+0000" 2024-04-16T07:43:43.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.381 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.381 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.381 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.382 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.382 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.889832+0000" 2024-04-16T07:43:43.382 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.382 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.382 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.382 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777305, 2024-04-16T07:43:43.382 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.382 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.382 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.851924+0000" 2024-04-16T07:43:43.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.382 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.383 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.383 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.851954+0000" 2024-04-16T07:43:43.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.383 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.383 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.383 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.852151+0000" 2024-04-16T07:43:43.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.383 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.383 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.383 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.383 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.383 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.383 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.852336+0000" 2024-04-16T07:43:43.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.384 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.384 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.384 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.854835+0000" 2024-04-16T07:43:43.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.384 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.384 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.384 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.384 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.384 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.952593+0000" 2024-04-16T07:43:43.384 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.384 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.385 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.385 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777309, 2024-04-16T07:43:43.385 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.385 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.385 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.951262+0000" 2024-04-16T07:43:43.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.385 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.385 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.385 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.951287+0000" 2024-04-16T07:43:43.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.385 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.386 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.386 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.951486+0000" 2024-04-16T07:43:43.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.386 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.386 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.386 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.386 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.386 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.386 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.951723+0000" 2024-04-16T07:43:43.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.386 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.386 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.386 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.955626+0000" 2024-04-16T07:43:43.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.387 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.387 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.387 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.387 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.387 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.045083+0000" 2024-04-16T07:43:43.387 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.387 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.387 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.387 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777310, 2024-04-16T07:43:43.387 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.388 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.388 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.981389+0000" 2024-04-16T07:43:43.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.388 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.388 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.388 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.981415+0000" 2024-04-16T07:43:43.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.388 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.388 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.388 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.981602+0000" 2024-04-16T07:43:43.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.388 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.388 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.389 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.389 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.389 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.389 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.981765+0000" 2024-04-16T07:43:43.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.389 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.389 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.389 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.984370+0000" 2024-04-16T07:43:43.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.389 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.389 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.389 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.390 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.390 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.078720+0000" 2024-04-16T07:43:43.390 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.390 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.390 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.390 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777313, 2024-04-16T07:43:43.390 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.390 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.390 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.045355+0000" 2024-04-16T07:43:43.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.391 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.391 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.391 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.045365+0000" 2024-04-16T07:43:43.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.391 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.391 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.391 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.045524+0000" 2024-04-16T07:43:43.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.391 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.391 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.391 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.391 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.391 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.392 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.045621+0000" 2024-04-16T07:43:43.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.392 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.392 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.392 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.047800+0000" 2024-04-16T07:43:43.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.392 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.392 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.392 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.392 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.392 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.143329+0000" 2024-04-16T07:43:43.392 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.392 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.393 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.393 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777315, 2024-04-16T07:43:43.393 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.393 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.393 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.056309+0000" 2024-04-16T07:43:43.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.393 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.393 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.393 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.056318+0000" 2024-04-16T07:43:43.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.393 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.393 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.394 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.056441+0000" 2024-04-16T07:43:43.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.394 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.394 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.394 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.394 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.394 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.394 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.056562+0000" 2024-04-16T07:43:43.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.394 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.394 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.394 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.057920+0000" 2024-04-16T07:43:43.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.394 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.395 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.395 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.395 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.395 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.156819+0000" 2024-04-16T07:43:43.395 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.395 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.395 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.395 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777316, 2024-04-16T07:43:43.395 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.395 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.395 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.063083+0000" 2024-04-16T07:43:43.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.396 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.396 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.396 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.063099+0000" 2024-04-16T07:43:43.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.396 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.396 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.396 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.063300+0000" 2024-04-16T07:43:43.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.396 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.396 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.396 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.396 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.396 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.396 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.063483+0000" 2024-04-16T07:43:43.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.397 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.397 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.397 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.065710+0000" 2024-04-16T07:43:43.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.397 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.397 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.397 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.397 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.397 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.179240+0000" 2024-04-16T07:43:43.397 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.397 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.398 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.398 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777318, 2024-04-16T07:43:43.398 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.398 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.398 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.067565+0000" 2024-04-16T07:43:43.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.398 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.398 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.398 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.067582+0000" 2024-04-16T07:43:43.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.398 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.398 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.399 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.067816+0000" 2024-04-16T07:43:43.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.399 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.399 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.399 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.399 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.399 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.399 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.067966+0000" 2024-04-16T07:43:43.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.399 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.399 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.399 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.072684+0000" 2024-04-16T07:43:43.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.399 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.399 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.400 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.400 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.400 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.195454+0000" 2024-04-16T07:43:43.400 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.400 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.400 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.400 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777324, 2024-04-16T07:43:43.400 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.400 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.400 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.150791+0000" 2024-04-16T07:43:43.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.401 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.401 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.401 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.150802+0000" 2024-04-16T07:43:43.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.401 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.401 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.401 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.150990+0000" 2024-04-16T07:43:43.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.401 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.401 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.401 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.401 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.401 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.402 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.151081+0000" 2024-04-16T07:43:43.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.402 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.402 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.402 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.151848+0000" 2024-04-16T07:43:43.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.402 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.402 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.402 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.402 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.402 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.246573+0000" 2024-04-16T07:43:43.402 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.402 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.403 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.403 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777326, 2024-04-16T07:43:43.403 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.403 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.403 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.177356+0000" 2024-04-16T07:43:43.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.403 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.403 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.403 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.177378+0000" 2024-04-16T07:43:43.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.403 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.403 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.404 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.177630+0000" 2024-04-16T07:43:43.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.404 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.404 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.404 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.404 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.404 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.404 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.177771+0000" 2024-04-16T07:43:43.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.404 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.404 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.404 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.179312+0000" 2024-04-16T07:43:43.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.404 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.404 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.405 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.405 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.405 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.279131+0000" 2024-04-16T07:43:43.405 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.405 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.405 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.405 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777329, 2024-04-16T07:43:43.405 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.405 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.405 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.256355+0000" 2024-04-16T07:43:43.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.406 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.406 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.406 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.256371+0000" 2024-04-16T07:43:43.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.406 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.406 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.406 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.256591+0000" 2024-04-16T07:43:43.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.406 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.406 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.406 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.406 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.406 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.406 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.256746+0000" 2024-04-16T07:43:43.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.407 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.407 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.407 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.257112+0000" 2024-04-16T07:43:43.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.407 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.407 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.407 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.407 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.407 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.407 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.407 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.407 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.407 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777269, 2024-04-16T07:43:43.408 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.408 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.408 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.361406+0000" 2024-04-16T07:43:43.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.408 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.408 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.408 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.361420+0000" 2024-04-16T07:43:43.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.408 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.408 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.408 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.361644+0000" 2024-04-16T07:43:43.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.408 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.408 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.409 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.409 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.409 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.409 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.361902+0000" 2024-04-16T07:43:43.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.409 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.409 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.409 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.362603+0000" 2024-04-16T07:43:43.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.409 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.409 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.409 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.410 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.410 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.367411+0000" 2024-04-16T07:43:43.410 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.410 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.410 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.410 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777271, 2024-04-16T07:43:43.410 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.410 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.410 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.363471+0000" 2024-04-16T07:43:43.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.410 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.411 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.411 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.363479+0000" 2024-04-16T07:43:43.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.411 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.411 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.411 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.363630+0000" 2024-04-16T07:43:43.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.411 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.411 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.411 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.411 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.411 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.411 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.363711+0000" 2024-04-16T07:43:43.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.412 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.412 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.412 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.364410+0000" 2024-04-16T07:43:43.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.412 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.412 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.412 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.412 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.412 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.372693+0000" 2024-04-16T07:43:43.412 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.412 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.413 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.413 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777273, 2024-04-16T07:43:43.413 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.413 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.413 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.365205+0000" 2024-04-16T07:43:43.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.413 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.413 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.413 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.365213+0000" 2024-04-16T07:43:43.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.413 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.413 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.413 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.365353+0000" 2024-04-16T07:43:43.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.414 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.414 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.414 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.414 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.414 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.414 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.365461+0000" 2024-04-16T07:43:43.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.414 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.414 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.414 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.368463+0000" 2024-04-16T07:43:43.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.414 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.415 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.415 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.415 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.415 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.374899+0000" 2024-04-16T07:43:43.415 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.415 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.415 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.415 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777272, 2024-04-16T07:43:43.415 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.416 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.416 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.364241+0000" 2024-04-16T07:43:43.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.416 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.416 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.416 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.364249+0000" 2024-04-16T07:43:43.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.416 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.416 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.416 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.364433+0000" 2024-04-16T07:43:43.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.416 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.416 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.416 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.417 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.417 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.417 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.364658+0000" 2024-04-16T07:43:43.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.417 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.417 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.417 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.367796+0000" 2024-04-16T07:43:43.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.417 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.417 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.417 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.417 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.417 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.489916+0000" 2024-04-16T07:43:43.417 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.418 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.418 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.418 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777276, 2024-04-16T07:43:43.418 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.418 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.418 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.367441+0000" 2024-04-16T07:43:43.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.418 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.418 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.418 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.367448+0000" 2024-04-16T07:43:43.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.418 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.419 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.419 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.367755+0000" 2024-04-16T07:43:43.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.419 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.419 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.419 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.419 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.419 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.419 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.367892+0000" 2024-04-16T07:43:43.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.419 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.419 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.419 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.368735+0000" 2024-04-16T07:43:43.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.420 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.420 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.420 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.420 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.420 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.492064+0000" 2024-04-16T07:43:43.420 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.420 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.420 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.420 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777277, 2024-04-16T07:43:43.420 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.421 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.421 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.368463+0000" 2024-04-16T07:43:43.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.421 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.421 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.421 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.368471+0000" 2024-04-16T07:43:43.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.421 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.421 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.421 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.368585+0000" 2024-04-16T07:43:43.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.421 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.421 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.421 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.422 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.422 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.422 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.368700+0000" 2024-04-16T07:43:43.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.422 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.422 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.422 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.373317+0000" 2024-04-16T07:43:43.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.422 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.422 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.422 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.423 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.423 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.493425+0000" 2024-04-16T07:43:43.423 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.423 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.423 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.423 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777278, 2024-04-16T07:43:43.423 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.423 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.423 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.369283+0000" 2024-04-16T07:43:43.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.423 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.423 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.424 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.369290+0000" 2024-04-16T07:43:43.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.424 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.424 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.424 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.369419+0000" 2024-04-16T07:43:43.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.424 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.424 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.424 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.424 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.424 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.424 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.369560+0000" 2024-04-16T07:43:43.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.424 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.425 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.425 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.374167+0000" 2024-04-16T07:43:43.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.425 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.425 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.425 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.425 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.425 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.495555+0000" 2024-04-16T07:43:43.425 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.425 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.426 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.426 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777285, 2024-04-16T07:43:43.426 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.426 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.426 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.465707+0000" 2024-04-16T07:43:43.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.426 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.426 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.426 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.465723+0000" 2024-04-16T07:43:43.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.426 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.426 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.426 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.465934+0000" 2024-04-16T07:43:43.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.427 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.427 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.427 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.427 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.427 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.427 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.466101+0000" 2024-04-16T07:43:43.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.427 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.427 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.427 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.466458+0000" 2024-04-16T07:43:43.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.427 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.427 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.428 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.428 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.428 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.515158+0000" 2024-04-16T07:43:43.428 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.428 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.428 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.428 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777286, 2024-04-16T07:43:43.428 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.428 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.428 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.471712+0000" 2024-04-16T07:43:43.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.429 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.429 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.429 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.471737+0000" 2024-04-16T07:43:43.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.429 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.429 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.429 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.471992+0000" 2024-04-16T07:43:43.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.429 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.429 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.429 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.429 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.430 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.430 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.472154+0000" 2024-04-16T07:43:43.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.430 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.430 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.430 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.472613+0000" 2024-04-16T07:43:43.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.430 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.430 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.430 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.430 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.430 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.537267+0000" 2024-04-16T07:43:43.430 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777287, 2024-04-16T07:43:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.497540+0000" 2024-04-16T07:43:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.497550+0000" 2024-04-16T07:43:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.497704+0000" 2024-04-16T07:43:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.497801+0000" 2024-04-16T07:43:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.498074+0000" 2024-04-16T07:43:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.557669+0000" 2024-04-16T07:43:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777289, 2024-04-16T07:43:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.513863+0000" 2024-04-16T07:43:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.513880+0000" 2024-04-16T07:43:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.514102+0000" 2024-04-16T07:43:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.514239+0000" 2024-04-16T07:43:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.515424+0000" 2024-04-16T07:43:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.577038+0000" 2024-04-16T07:43:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777290, 2024-04-16T07:43:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.542544+0000" 2024-04-16T07:43:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.542567+0000" 2024-04-16T07:43:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.543033+0000" 2024-04-16T07:43:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.543187+0000" 2024-04-16T07:43:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.545401+0000" 2024-04-16T07:43:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.628407+0000" 2024-04-16T07:43:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777291, 2024-04-16T07:43:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.545904+0000" 2024-04-16T07:43:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.545914+0000" 2024-04-16T07:43:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.546071+0000" 2024-04-16T07:43:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.546187+0000" 2024-04-16T07:43:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.546669+0000" 2024-04-16T07:43:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.650001+0000" 2024-04-16T07:43:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777295, 2024-04-16T07:43:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.619312+0000" 2024-04-16T07:43:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.619324+0000" 2024-04-16T07:43:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.619479+0000" 2024-04-16T07:43:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.619570+0000" 2024-04-16T07:43:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.619880+0000" 2024-04-16T07:43:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.709596+0000" 2024-04-16T07:43:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777301, 2024-04-16T07:43:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.741451+0000" 2024-04-16T07:43:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.741467+0000" 2024-04-16T07:43:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.741660+0000" 2024-04-16T07:43:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.741839+0000" 2024-04-16T07:43:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.742186+0000" 2024-04-16T07:43:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.807153+0000" 2024-04-16T07:43:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777302, 2024-04-16T07:43:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.779279+0000" 2024-04-16T07:43:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.779295+0000" 2024-04-16T07:43:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.779454+0000" 2024-04-16T07:43:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.779566+0000" 2024-04-16T07:43:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.779832+0000" 2024-04-16T07:43:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.848230+0000" 2024-04-16T07:43:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777304, 2024-04-16T07:43:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.799395+0000" 2024-04-16T07:43:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.799405+0000" 2024-04-16T07:43:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.799545+0000" 2024-04-16T07:43:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.799629+0000" 2024-04-16T07:43:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.799931+0000" 2024-04-16T07:43:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.883514+0000" 2024-04-16T07:43:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777306, 2024-04-16T07:43:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.895670+0000" 2024-04-16T07:43:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.895682+0000" 2024-04-16T07:43:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.895854+0000" 2024-04-16T07:43:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.896006+0000" 2024-04-16T07:43:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.897211+0000" 2024-04-16T07:43:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.938525+0000" 2024-04-16T07:43:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777307, 2024-04-16T07:43:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.906516+0000" 2024-04-16T07:43:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.906540+0000" 2024-04-16T07:43:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.906735+0000" 2024-04-16T07:43:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.906824+0000" 2024-04-16T07:43:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.907062+0000" 2024-04-16T07:43:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.958014+0000" 2024-04-16T07:43:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777308, 2024-04-16T07:43:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.944511+0000" 2024-04-16T07:43:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.944526+0000" 2024-04-16T07:43:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.944727+0000" 2024-04-16T07:43:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.944873+0000" 2024-04-16T07:43:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.945211+0000" 2024-04-16T07:43:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.033773+0000" 2024-04-16T07:43:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777311, 2024-04-16T07:43:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.982157+0000" 2024-04-16T07:43:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.982164+0000" 2024-04-16T07:43:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.982296+0000" 2024-04-16T07:43:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.982382+0000" 2024-04-16T07:43:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:42.982580+0000" 2024-04-16T07:43:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.053163+0000" 2024-04-16T07:43:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777312, 2024-04-16T07:43:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.021602+0000" 2024-04-16T07:43:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.021613+0000" 2024-04-16T07:43:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.021843+0000" 2024-04-16T07:43:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.021969+0000" 2024-04-16T07:43:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.023783+0000" 2024-04-16T07:43:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.073347+0000" 2024-04-16T07:43:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777314, 2024-04-16T07:43:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.055442+0000" 2024-04-16T07:43:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.055454+0000" 2024-04-16T07:43:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.055579+0000" 2024-04-16T07:43:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.055669+0000" 2024-04-16T07:43:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.055921+0000" 2024-04-16T07:43:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.124491+0000" 2024-04-16T07:43:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777317, 2024-04-16T07:43:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.064072+0000" 2024-04-16T07:43:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.064082+0000" 2024-04-16T07:43:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.064233+0000" 2024-04-16T07:43:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.064414+0000" 2024-04-16T07:43:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.064750+0000" 2024-04-16T07:43:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.159282+0000" 2024-04-16T07:43:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777320, 2024-04-16T07:43:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.076680+0000" 2024-04-16T07:43:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.076692+0000" 2024-04-16T07:43:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.076890+0000" 2024-04-16T07:43:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.077027+0000" 2024-04-16T07:43:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.077640+0000" 2024-04-16T07:43:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.159583+0000" 2024-04-16T07:43:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777321, 2024-04-16T07:43:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.096067+0000" 2024-04-16T07:43:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.096093+0000" 2024-04-16T07:43:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.096343+0000" 2024-04-16T07:43:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.096494+0000" 2024-04-16T07:43:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.096849+0000" 2024-04-16T07:43:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.163742+0000" 2024-04-16T07:43:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777319, 2024-04-16T07:43:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.074728+0000" 2024-04-16T07:43:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.074744+0000" 2024-04-16T07:43:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.074930+0000" 2024-04-16T07:43:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.075122+0000" 2024-04-16T07:43:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.075715+0000" 2024-04-16T07:43:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.182653+0000" 2024-04-16T07:43:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777322, 2024-04-16T07:43:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.125585+0000" 2024-04-16T07:43:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.125595+0000" 2024-04-16T07:43:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.125760+0000" 2024-04-16T07:43:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.125845+0000" 2024-04-16T07:43:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.126109+0000" 2024-04-16T07:43:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.217383+0000" 2024-04-16T07:43:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777323, 2024-04-16T07:43:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.146762+0000" 2024-04-16T07:43:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.146779+0000" 2024-04-16T07:43:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.146954+0000" 2024-04-16T07:43:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.147048+0000" 2024-04-16T07:43:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.147295+0000" 2024-04-16T07:43:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.226124+0000" 2024-04-16T07:43:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777325, 2024-04-16T07:43:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.161210+0000" 2024-04-16T07:43:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.161230+0000" 2024-04-16T07:43:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.161371+0000" 2024-04-16T07:43:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.161475+0000" 2024-04-16T07:43:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.161705+0000" 2024-04-16T07:43:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.260179+0000" 2024-04-16T07:43:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:43:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:43:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16777327, 2024-04-16T07:43:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:43:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:43:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.179863+0000" 2024-04-16T07:43:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:43:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.179873+0000" 2024-04-16T07:43:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:43:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.180008+0000" 2024-04-16T07:43:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:43:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:43:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.180160+0000" 2024-04-16T07:43:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:43:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.180384+0000" 2024-04-16T07:43:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:43:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:43:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:43:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:43:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:43:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:43:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:43:43.279678+0000" 2024-04-16T07:43:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:43:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-04-16T07:43:43.485 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-16T07:43:43.485 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 45 16 4180 4164 5.78262 12.1875 0.115303 0.172788 2024-04-16T07:43:44.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:43:44.397 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 46 16 4396 4380 5.95036 13.5 0.0776193 0.167919 2024-04-16T07:43:44.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:44.499+0000 7f797ed9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:44.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:44.508+0000 7f797ed9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:45.397 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 47 16 4585 4569 6.07506 11.8125 0.120005 0.164362 2024-04-16T07:43:46.398 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 48 16 4789 4773 6.21409 12.75 0.101098 0.160726 2024-04-16T07:43:46.838 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:43:47.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:46.999+0000 7f8cc84ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:47.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:47.010+0000 7f8cc84ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:47.398 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 49 16 4986 4970 6.33852 12.3125 0.130239 0.157564 2024-04-16T07:43:48.398 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 50 16 5186 5170 6.46173 12.5 0.0351824 0.154614 2024-04-16T07:43:48.487 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-16T07:43:48.487 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:43:48.487 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3] 2024-04-16T07:43:48.487 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-16T07:43:48.487 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:43:48.555 INFO:tasks.ceph.osd.1:Stopped 2024-04-16T07:43:49.398 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 51 16 5212 5196 6.36688 1.625 0.0323223 0.154259 2024-04-16T07:43:49.432 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:43:49.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:49.558+0000 7fd3ce19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:49.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:43:49.564+0000 7fd3ce19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:43:50.398 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 52 16 5212 5196 6.24444 0 - 0.154259 2024-04-16T07:43:51.398 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 53 16 5212 5196 6.12662 0 - 0.154259 2024-04-16T07:43:52.398 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 54 16 5212 5196 6.01316 0 - 0.154259 2024-04-16T07:43:53.399 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 55 16 5212 5196 5.90383 0 - 0.154259 2024-04-16T07:43:53.556 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3] 2024-04-16T07:43:53.556 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:43:53.556 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-04-16T07:43:53.556 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-16T07:43:53.556 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-16T07:43:53.559 INFO:tasks.ceph.osd.1:Started 2024-04-16T07:43:53.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-16T07:43:53.677 INFO:teuthology.orchestra.run.smithi031.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:43:53.680 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:43:53.680 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-16T07:43:54.399 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 56 16 5212 5196 5.79841 0 - 0.154259 2024-04-16T07:43:55.399 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 57 16 5212 5196 5.69668 0 - 0.154259 2024-04-16T07:43:56.399 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 58 16 5212 5196 5.59847 0 - 0.154259 2024-04-16T07:43:57.399 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 59 16 5212 5196 5.50358 0 - 0.154259 2024-04-16T07:43:58.399 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:2024-04-16T07:43:58.400766+0000 min lat: 0.0168854 max lat: 25.1577 avg lat: 0.154259 2024-04-16T07:43:58.399 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:43:58.399 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 60 16 5212 5196 5.41186 0 - 0.154259 2024-04-16T07:43:58.681 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-16T07:43:58.786 INFO:teuthology.orchestra.run.smithi031.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:43:58.789 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:43:58.789 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-16T07:43:59.399 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 61 16 5212 5196 5.32314 0 - 0.154259 2024-04-16T07:44:00.399 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 62 16 5212 5196 5.23728 0 - 0.154259 2024-04-16T07:44:01.399 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 63 16 5212 5196 5.15415 0 - 0.154259 2024-04-16T07:44:02.399 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 64 16 5212 5196 5.07362 0 - 0.154259 2024-04-16T07:44:03.399 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 65 16 5212 5196 4.99556 0 - 0.154259 2024-04-16T07:44:03.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-16T07:44:03.898 INFO:teuthology.orchestra.run.smithi031.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:44:03.901 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:44:03.902 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-16T07:44:04.400 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 66 16 5212 5196 4.91988 0 - 0.154259 2024-04-16T07:44:05.400 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 67 16 5212 5196 4.84645 0 - 0.154259 2024-04-16T07:44:05.924 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-16T07:44:06.079 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:44:06.080 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:44:06.400 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 68 16 5212 5196 4.77517 0 - 0.154259 2024-04-16T07:44:07.400 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 69 16 5212 5196 4.70597 0 - 0.154259 2024-04-16T07:44:08.081 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:44:08.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:08.229+0000 7f57980ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:08.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:08.239+0000 7f57980ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:08.400 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 70 16 5236 5220 4.66016 0.0789474 19.8412 0.214354 2024-04-16T07:44:08.903 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-16T07:44:09.115 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-16T07:44:09.115 INFO:teuthology.orchestra.run.smithi031.stdout: "ops_in_flight": [ 2024-04-16T07:44:09.115 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.115 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.115 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 312, 2024-04-16T07:44:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.059879+0000" 2024-04-16T07:44:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.059901+0000" 2024-04-16T07:44:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.064578+0000" 2024-04-16T07:44:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.068103+0000" 2024-04-16T07:44:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.068541+0000" 2024-04-16T07:44:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.118 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.118 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.118 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.118 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.118 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.118 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.118 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 315, 2024-04-16T07:44:09.118 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.118 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.118 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.118 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.096105+0000" 2024-04-16T07:44:09.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.118 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.096117+0000" 2024-04-16T07:44:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.096405+0000", 2024-04-16T07:44:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: "blocker": { 2024-04-16T07:44:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: "op_type": "ConnectionPipeline::await_map", 2024-04-16T07:44:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: "waiting": 0, 2024-04-16T07:44:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: "held_by_operation_id": 315 2024-04-16T07:44:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 190, 2024-04-16T07:44:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.276341+0000" 2024-04-16T07:44:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.276354+0000" 2024-04-16T07:44:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.277749+0000" 2024-04-16T07:44:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.279373+0000" 2024-04-16T07:44:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.280134+0000" 2024-04-16T07:44:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.294052+0000" 2024-04-16T07:44:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 198, 2024-04-16T07:44:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.307443+0000" 2024-04-16T07:44:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.307469+0000" 2024-04-16T07:44:09.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.125 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.125 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.125 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.310147+0000" 2024-04-16T07:44:09.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.125 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.125 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.125 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.125 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.125 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.125 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.310346+0000" 2024-04-16T07:44:09.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.125 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.125 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.125 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.310627+0000" 2024-04-16T07:44:09.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.126 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.126 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.126 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.126 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.126 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.398368+0000" 2024-04-16T07:44:09.126 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.126 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.126 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.126 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 209, 2024-04-16T07:44:09.126 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.127 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.127 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.372554+0000" 2024-04-16T07:44:09.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.127 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.127 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.127 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.372571+0000" 2024-04-16T07:44:09.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.127 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.127 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.127 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.376480+0000" 2024-04-16T07:44:09.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.127 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.127 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.127 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.128 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.128 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.128 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.379201+0000" 2024-04-16T07:44:09.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.128 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.128 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.128 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.379493+0000" 2024-04-16T07:44:09.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.128 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.128 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.128 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.128 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.128 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.490118+0000" 2024-04-16T07:44:09.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.129 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.129 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 236, 2024-04-16T07:44:09.129 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.129 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.129 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.129 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.560855+0000" 2024-04-16T07:44:09.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.129 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.129 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.129 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.560870+0000" 2024-04-16T07:44:09.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.129 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.130 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.130 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.561027+0000" 2024-04-16T07:44:09.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.130 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.130 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.130 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.130 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.130 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.130 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.561357+0000" 2024-04-16T07:44:09.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.130 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.130 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.130 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.561548+0000" 2024-04-16T07:44:09.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.131 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.131 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.131 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.131 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.131 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.629212+0000" 2024-04-16T07:44:09.131 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.131 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.131 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.131 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 247, 2024-04-16T07:44:09.132 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.132 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.132 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.630076+0000" 2024-04-16T07:44:09.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.132 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.132 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.132 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.630090+0000" 2024-04-16T07:44:09.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.132 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.132 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.132 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.630309+0000" 2024-04-16T07:44:09.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.132 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.132 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.133 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.133 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.133 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.133 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.630467+0000" 2024-04-16T07:44:09.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.133 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.133 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.133 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.630633+0000" 2024-04-16T07:44:09.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.133 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.133 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.133 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.134 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.134 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.708366+0000" 2024-04-16T07:44:09.134 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.134 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.134 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.134 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 264, 2024-04-16T07:44:09.134 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.134 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.134 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.732598+0000" 2024-04-16T07:44:09.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.134 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.134 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.135 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.732613+0000" 2024-04-16T07:44:09.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.135 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.135 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.135 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.732812+0000" 2024-04-16T07:44:09.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.135 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.135 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.135 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.135 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.135 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.136 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.732909+0000" 2024-04-16T07:44:09.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.136 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.136 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.136 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.734372+0000" 2024-04-16T07:44:09.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.136 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.136 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.136 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.136 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.136 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.798233+0000" 2024-04-16T07:44:09.136 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.137 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.137 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.137 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 266, 2024-04-16T07:44:09.137 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.137 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.137 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.739441+0000" 2024-04-16T07:44:09.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.137 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.137 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.137 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.739457+0000" 2024-04-16T07:44:09.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.137 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.138 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.138 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.740156+0000" 2024-04-16T07:44:09.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.138 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.138 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.138 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.138 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.138 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.138 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.740809+0000" 2024-04-16T07:44:09.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.138 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.139 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.139 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.741719+0000" 2024-04-16T07:44:09.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.139 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.139 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.139 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.139 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.139 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.824961+0000" 2024-04-16T07:44:09.139 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.139 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.139 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.139 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 285, 2024-04-16T07:44:09.140 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.140 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.140 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.854821+0000" 2024-04-16T07:44:09.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.140 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.140 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.140 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.854832+0000" 2024-04-16T07:44:09.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.140 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.140 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.140 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.854988+0000" 2024-04-16T07:44:09.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.141 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.141 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.141 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.141 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.141 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.141 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.855388+0000" 2024-04-16T07:44:09.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.141 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.141 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.141 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.855578+0000" 2024-04-16T07:44:09.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.141 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.141 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.141 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.142 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.142 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.921765+0000" 2024-04-16T07:44:09.142 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.142 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.142 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.142 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 302, 2024-04-16T07:44:09.142 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.142 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.142 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.978901+0000" 2024-04-16T07:44:09.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.978929+0000" 2024-04-16T07:44:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.979356+0000" 2024-04-16T07:44:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.979588+0000" 2024-04-16T07:44:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.979845+0000" 2024-04-16T07:44:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.094389+0000" 2024-04-16T07:44:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 181, 2024-04-16T07:44:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.266314+0000" 2024-04-16T07:44:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.266337+0000" 2024-04-16T07:44:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.266473+0000" 2024-04-16T07:44:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.146 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.146 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.146 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.146 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.266623+0000" 2024-04-16T07:44:09.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.146 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.146 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.146 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.267393+0000" 2024-04-16T07:44:09.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.146 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.146 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.146 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.273047+0000" 2024-04-16T07:44:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 197, 2024-04-16T07:44:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.300286+0000" 2024-04-16T07:44:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.300302+0000" 2024-04-16T07:44:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.301066+0000" 2024-04-16T07:44:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.303223+0000" 2024-04-16T07:44:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.304453+0000" 2024-04-16T07:44:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.369042+0000" 2024-04-16T07:44:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.150 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 200, 2024-04-16T07:44:09.150 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.150 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.150 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.323222+0000" 2024-04-16T07:44:09.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.150 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.150 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.150 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.323236+0000" 2024-04-16T07:44:09.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.150 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.150 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.150 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.323857+0000" 2024-04-16T07:44:09.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.324013+0000" 2024-04-16T07:44:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.328224+0000" 2024-04-16T07:44:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.395501+0000" 2024-04-16T07:44:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 202, 2024-04-16T07:44:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.153 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.335642+0000" 2024-04-16T07:44:09.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.153 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.153 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.153 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.335658+0000" 2024-04-16T07:44:09.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.153 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.153 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.153 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.340891+0000" 2024-04-16T07:44:09.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.153 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.153 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.153 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.154 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.154 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.154 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.341223+0000" 2024-04-16T07:44:09.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.154 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.154 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.154 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.345803+0000" 2024-04-16T07:44:09.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.154 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.154 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.154 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.155 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.155 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.155 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.411102+0000" 2024-04-16T07:44:09.155 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.155 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.155 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.155 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 204, 2024-04-16T07:44:09.155 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.155 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.155 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.155 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.349286+0000" 2024-04-16T07:44:09.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.155 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.156 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.156 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.349295+0000" 2024-04-16T07:44:09.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.156 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.156 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.156 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.349753+0000" 2024-04-16T07:44:09.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.156 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.156 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.156 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.156 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.156 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.157 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.349911+0000" 2024-04-16T07:44:09.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.157 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.157 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.157 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.352425+0000" 2024-04-16T07:44:09.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.157 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.157 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.157 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.157 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.157 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.423058+0000" 2024-04-16T07:44:09.158 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.158 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.158 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.158 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 206, 2024-04-16T07:44:09.158 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.158 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.158 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.353417+0000" 2024-04-16T07:44:09.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.158 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.158 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.158 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.353432+0000" 2024-04-16T07:44:09.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.159 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.159 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.159 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.355422+0000" 2024-04-16T07:44:09.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.159 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.159 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.159 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.159 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.159 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.159 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.356049+0000" 2024-04-16T07:44:09.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.159 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.160 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.160 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.361690+0000" 2024-04-16T07:44:09.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.160 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.160 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.160 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.160 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.160 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.435599+0000" 2024-04-16T07:44:09.160 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.160 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.160 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.160 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 210, 2024-04-16T07:44:09.161 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.161 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.161 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.385342+0000" 2024-04-16T07:44:09.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.161 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.161 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.161 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.385358+0000" 2024-04-16T07:44:09.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.161 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.161 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.161 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.385546+0000" 2024-04-16T07:44:09.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.161 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.161 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.162 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.162 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.162 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.162 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.385819+0000" 2024-04-16T07:44:09.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.162 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.162 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.162 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.386984+0000" 2024-04-16T07:44:09.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.162 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.162 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.162 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.163 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.163 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.465852+0000" 2024-04-16T07:44:09.163 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.163 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.163 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.163 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 211, 2024-04-16T07:44:09.163 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.163 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.163 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.394725+0000" 2024-04-16T07:44:09.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.164 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.164 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.164 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.394741+0000" 2024-04-16T07:44:09.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.164 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.164 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.164 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.395606+0000" 2024-04-16T07:44:09.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.164 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.164 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.164 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.164 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.164 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.165 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.397533+0000" 2024-04-16T07:44:09.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.165 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.165 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.165 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.401436+0000" 2024-04-16T07:44:09.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.165 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.165 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.165 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.165 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.165 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.496464+0000" 2024-04-16T07:44:09.166 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.166 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.166 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.166 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 232, 2024-04-16T07:44:09.166 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.166 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.166 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.536739+0000" 2024-04-16T07:44:09.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.166 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.166 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.166 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.536755+0000" 2024-04-16T07:44:09.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.167 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.167 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.167 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.537035+0000" 2024-04-16T07:44:09.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.167 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.167 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.167 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.167 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.167 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.167 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.537245+0000" 2024-04-16T07:44:09.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.168 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.168 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.168 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.539494+0000" 2024-04-16T07:44:09.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.168 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.168 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.168 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.168 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.168 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.616173+0000" 2024-04-16T07:44:09.168 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.168 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.169 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.169 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 235, 2024-04-16T07:44:09.169 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.169 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.169 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.550969+0000" 2024-04-16T07:44:09.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.169 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.169 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.169 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.550993+0000" 2024-04-16T07:44:09.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.169 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.169 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.169 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.553140+0000" 2024-04-16T07:44:09.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.170 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.170 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.170 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.170 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.170 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.170 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.554859+0000" 2024-04-16T07:44:09.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.170 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.170 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.170 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.555604+0000" 2024-04-16T07:44:09.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.170 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.170 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.171 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.171 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.171 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.623350+0000" 2024-04-16T07:44:09.171 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.171 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.171 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.171 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 246, 2024-04-16T07:44:09.172 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.172 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.172 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.625139+0000" 2024-04-16T07:44:09.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.172 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.172 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.172 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.625154+0000" 2024-04-16T07:44:09.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.172 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.172 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.172 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.627436+0000" 2024-04-16T07:44:09.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.172 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.172 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.173 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.173 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.173 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.173 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.627964+0000" 2024-04-16T07:44:09.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.173 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.173 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.173 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.628645+0000" 2024-04-16T07:44:09.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.173 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.173 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.173 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.174 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.174 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.174 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.720333+0000" 2024-04-16T07:44:09.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.174 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.174 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 258, 2024-04-16T07:44:09.174 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.174 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.174 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.175 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.687802+0000" 2024-04-16T07:44:09.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.175 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.175 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.175 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.687823+0000" 2024-04-16T07:44:09.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.175 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.175 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.175 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.688692+0000" 2024-04-16T07:44:09.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.175 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.175 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.175 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.176 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.176 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.176 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.693581+0000" 2024-04-16T07:44:09.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.176 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.176 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.176 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.694454+0000" 2024-04-16T07:44:09.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.176 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.176 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.176 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.176 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.176 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.772820+0000" 2024-04-16T07:44:09.176 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.177 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.177 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.177 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 261, 2024-04-16T07:44:09.177 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.177 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.177 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.708983+0000" 2024-04-16T07:44:09.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.177 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.177 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.177 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.708993+0000" 2024-04-16T07:44:09.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.177 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.178 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.178 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.709502+0000" 2024-04-16T07:44:09.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.178 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.178 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.178 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.178 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.178 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.178 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.710082+0000" 2024-04-16T07:44:09.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.178 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.178 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.178 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.711331+0000" 2024-04-16T07:44:09.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.179 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.179 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.179 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.179 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.179 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.794665+0000" 2024-04-16T07:44:09.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.179 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.179 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 270, 2024-04-16T07:44:09.179 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.180 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.180 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.767377+0000" 2024-04-16T07:44:09.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.180 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.180 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.180 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.767396+0000" 2024-04-16T07:44:09.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.180 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.180 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.180 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.767856+0000" 2024-04-16T07:44:09.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.180 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.180 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.180 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.181 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.181 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.181 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.768114+0000" 2024-04-16T07:44:09.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.181 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.181 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.181 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.769291+0000" 2024-04-16T07:44:09.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.181 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.181 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.181 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.181 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.181 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.843416+0000" 2024-04-16T07:44:09.182 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.182 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.182 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.182 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 274, 2024-04-16T07:44:09.182 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.182 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.182 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.798341+0000" 2024-04-16T07:44:09.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.182 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.182 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.183 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.798353+0000" 2024-04-16T07:44:09.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.183 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.183 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.183 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.798790+0000" 2024-04-16T07:44:09.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.183 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.183 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.183 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.183 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.183 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.183 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.799638+0000" 2024-04-16T07:44:09.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.184 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.184 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.184 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.800177+0000" 2024-04-16T07:44:09.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.184 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.184 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.184 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.184 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.184 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.863987+0000" 2024-04-16T07:44:09.184 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.184 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.185 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.185 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 288, 2024-04-16T07:44:09.185 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.185 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.185 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.877988+0000" 2024-04-16T07:44:09.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.185 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.185 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.185 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.877997+0000" 2024-04-16T07:44:09.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.185 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.878181+0000" 2024-04-16T07:44:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.878276+0000" 2024-04-16T07:44:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.881332+0000" 2024-04-16T07:44:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.187 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.187 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.187 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.187 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.187 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.956346+0000" 2024-04-16T07:44:09.187 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.187 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.187 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.187 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 314, 2024-04-16T07:44:09.187 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.188 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.188 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.079194+0000" 2024-04-16T07:44:09.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.188 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.188 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.188 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.079214+0000" 2024-04-16T07:44:09.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.188 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.188 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.188 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.080668+0000" 2024-04-16T07:44:09.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.188 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.188 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.188 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.189 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.189 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.189 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.081946+0000" 2024-04-16T07:44:09.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.189 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.189 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.189 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.091865+0000" 2024-04-16T07:44:09.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.189 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.189 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.189 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.190 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.190 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.190 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.190 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.190 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.190 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 183, 2024-04-16T07:44:09.190 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.191 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.191 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.268178+0000" 2024-04-16T07:44:09.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.191 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.191 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.191 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.268187+0000" 2024-04-16T07:44:09.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.191 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.191 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.191 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.269021+0000" 2024-04-16T07:44:09.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.191 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.191 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.191 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.192 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.192 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.192 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.269346+0000" 2024-04-16T07:44:09.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.192 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.192 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.192 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.270682+0000" 2024-04-16T07:44:09.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.192 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.192 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.192 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.192 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.192 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.193 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.276394+0000" 2024-04-16T07:44:09.193 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.193 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.193 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.193 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 187, 2024-04-16T07:44:09.193 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.193 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.194 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.194 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.272573+0000" 2024-04-16T07:44:09.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.194 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.194 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.194 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.272583+0000" 2024-04-16T07:44:09.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.194 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.194 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.194 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.272893+0000" 2024-04-16T07:44:09.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.194 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.194 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.195 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.195 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.195 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.195 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.273101+0000" 2024-04-16T07:44:09.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.195 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.195 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.195 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.274075+0000" 2024-04-16T07:44:09.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.195 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.195 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.195 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.196 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.196 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.278628+0000" 2024-04-16T07:44:09.196 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.196 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.196 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.196 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 192, 2024-04-16T07:44:09.196 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.196 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.196 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.287863+0000" 2024-04-16T07:44:09.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.196 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.197 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.197 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.287877+0000" 2024-04-16T07:44:09.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.197 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.197 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.197 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.289758+0000" 2024-04-16T07:44:09.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.197 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.197 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.197 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.197 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.197 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.197 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.290076+0000" 2024-04-16T07:44:09.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.198 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.198 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.198 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.292608+0000" 2024-04-16T07:44:09.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.198 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.198 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.198 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.198 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.198 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.198 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.357616+0000" 2024-04-16T07:44:09.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.199 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.199 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 194, 2024-04-16T07:44:09.199 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.199 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.199 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.199 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.295697+0000" 2024-04-16T07:44:09.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.199 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.199 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.199 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.295721+0000" 2024-04-16T07:44:09.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.199 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.199 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.200 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.296244+0000" 2024-04-16T07:44:09.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.200 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.200 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.200 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.200 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.200 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.200 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.296579+0000" 2024-04-16T07:44:09.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.200 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.200 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.201 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.301039+0000" 2024-04-16T07:44:09.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.201 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.201 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.201 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.201 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.201 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.386722+0000" 2024-04-16T07:44:09.201 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.201 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.201 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.202 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 207, 2024-04-16T07:44:09.202 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.202 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.202 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.364401+0000" 2024-04-16T07:44:09.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.202 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.202 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.202 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.364411+0000" 2024-04-16T07:44:09.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.202 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.202 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.202 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.364590+0000" 2024-04-16T07:44:09.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.203 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.203 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.203 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.203 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.203 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.203 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.366132+0000" 2024-04-16T07:44:09.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.203 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.203 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.203 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.367194+0000" 2024-04-16T07:44:09.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.203 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.204 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.204 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.204 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.204 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.204 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.457500+0000" 2024-04-16T07:44:09.204 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.204 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.204 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.204 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 212, 2024-04-16T07:44:09.204 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.204 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.204 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.205 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.403763+0000" 2024-04-16T07:44:09.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.205 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.205 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.205 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.403773+0000" 2024-04-16T07:44:09.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.205 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.205 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.205 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.404335+0000" 2024-04-16T07:44:09.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.205 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.205 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.205 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.206 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.206 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.206 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.404731+0000" 2024-04-16T07:44:09.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.206 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.206 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.206 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.405761+0000" 2024-04-16T07:44:09.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.206 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.206 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.207 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.207 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.207 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.207 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.503792+0000" 2024-04-16T07:44:09.207 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.207 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.207 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.207 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 215, 2024-04-16T07:44:09.207 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.207 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.207 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.208 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.406592+0000" 2024-04-16T07:44:09.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.208 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.208 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.208 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.406606+0000" 2024-04-16T07:44:09.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.208 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.208 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.208 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.407191+0000" 2024-04-16T07:44:09.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.208 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.208 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.208 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.209 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.209 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.209 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.407704+0000" 2024-04-16T07:44:09.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.209 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.209 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.209 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.413388+0000" 2024-04-16T07:44:09.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.209 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.209 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.209 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.209 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.210 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.210 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.519988+0000" 2024-04-16T07:44:09.210 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.210 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.210 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.210 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.210 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 217, 2024-04-16T07:44:09.210 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.210 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.210 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.210 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.452177+0000" 2024-04-16T07:44:09.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.211 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.211 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.211 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.452204+0000" 2024-04-16T07:44:09.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.211 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.211 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.211 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.452588+0000" 2024-04-16T07:44:09.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.211 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.211 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.211 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.211 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.212 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.212 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.452797+0000" 2024-04-16T07:44:09.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.212 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.212 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.212 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.453901+0000" 2024-04-16T07:44:09.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.212 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.212 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.212 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.212 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.212 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.212 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.538345+0000" 2024-04-16T07:44:09.213 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.213 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.213 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.213 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 220, 2024-04-16T07:44:09.213 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.213 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.213 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.213 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.458736+0000" 2024-04-16T07:44:09.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.213 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.213 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.213 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.458748+0000" 2024-04-16T07:44:09.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.214 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.214 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.214 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.458981+0000" 2024-04-16T07:44:09.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.214 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.214 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.214 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.214 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.214 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.214 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.459316+0000" 2024-04-16T07:44:09.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.214 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.215 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.215 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.461857+0000" 2024-04-16T07:44:09.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.215 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.215 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.215 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.215 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.215 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.215 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.545722+0000" 2024-04-16T07:44:09.215 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.215 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.216 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.216 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.216 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 226, 2024-04-16T07:44:09.216 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.216 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.216 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.216 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.499482+0000" 2024-04-16T07:44:09.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.216 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.216 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.216 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.499495+0000" 2024-04-16T07:44:09.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.216 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.217 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.217 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.499703+0000" 2024-04-16T07:44:09.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.217 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.217 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.217 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.217 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.217 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.217 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.500300+0000" 2024-04-16T07:44:09.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.217 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.218 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.218 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.501005+0000" 2024-04-16T07:44:09.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.218 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.218 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.218 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.218 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.218 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.218 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.597094+0000" 2024-04-16T07:44:09.218 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.218 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.219 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.219 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 263, 2024-04-16T07:44:09.219 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.219 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.219 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.219 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.715202+0000" 2024-04-16T07:44:09.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.219 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.219 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.219 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.715217+0000" 2024-04-16T07:44:09.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.219 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.220 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.220 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.716219+0000" 2024-04-16T07:44:09.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.220 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.220 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.220 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.220 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.220 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.220 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.716480+0000" 2024-04-16T07:44:09.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.220 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.220 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.220 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.717074+0000" 2024-04-16T07:44:09.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.221 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.221 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.221 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.221 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.221 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.221 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.787316+0000" 2024-04-16T07:44:09.221 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.221 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.221 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.221 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 269, 2024-04-16T07:44:09.221 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.222 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.222 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.222 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.755314+0000" 2024-04-16T07:44:09.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.222 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.222 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.222 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.755324+0000" 2024-04-16T07:44:09.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.222 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.222 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.222 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.756472+0000" 2024-04-16T07:44:09.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.222 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.223 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.223 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.223 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.223 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.223 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.757599+0000" 2024-04-16T07:44:09.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.223 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.223 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.223 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.761427+0000" 2024-04-16T07:44:09.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.223 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.223 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.223 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.224 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.224 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.224 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.845829+0000" 2024-04-16T07:44:09.224 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.224 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.224 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.224 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 284, 2024-04-16T07:44:09.224 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.224 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.224 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.225 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.845394+0000" 2024-04-16T07:44:09.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.225 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.225 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.225 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.845405+0000" 2024-04-16T07:44:09.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.225 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.225 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.225 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.845553+0000" 2024-04-16T07:44:09.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.225 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.225 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.225 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.226 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.226 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.226 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.845649+0000" 2024-04-16T07:44:09.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.226 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.226 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.226 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.846173+0000" 2024-04-16T07:44:09.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.226 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.226 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.226 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.226 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.227 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.227 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.917536+0000" 2024-04-16T07:44:09.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.227 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.227 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 290, 2024-04-16T07:44:09.227 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.227 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.227 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.227 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.883867+0000" 2024-04-16T07:44:09.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.227 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.228 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.228 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.883891+0000" 2024-04-16T07:44:09.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.228 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.228 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.228 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.884184+0000" 2024-04-16T07:44:09.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.228 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.228 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.228 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.228 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.228 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.228 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.884443+0000" 2024-04-16T07:44:09.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.229 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.229 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.229 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.888917+0000" 2024-04-16T07:44:09.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.229 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.229 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.229 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.229 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.229 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.229 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.962161+0000" 2024-04-16T07:44:09.229 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.229 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.230 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.230 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 297, 2024-04-16T07:44:09.230 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.230 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.230 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.230 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.943870+0000" 2024-04-16T07:44:09.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.230 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.230 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.230 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.943888+0000" 2024-04-16T07:44:09.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.230 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.230 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.231 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.944506+0000" 2024-04-16T07:44:09.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.231 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.231 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.231 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.231 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.231 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.231 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.944852+0000" 2024-04-16T07:44:09.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.231 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.231 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.231 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:08.945436+0000" 2024-04-16T07:44:09.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.231 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.232 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.232 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.232 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.232 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.232 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.009909+0000" 2024-04-16T07:44:09.232 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.232 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.232 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.232 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 305, 2024-04-16T07:44:09.233 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.233 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.233 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.008548+0000" 2024-04-16T07:44:09.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.233 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.233 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.233 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.008574+0000" 2024-04-16T07:44:09.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.233 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.233 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.233 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.012488+0000" 2024-04-16T07:44:09.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.233 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.233 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.234 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.234 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.234 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.234 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.013385+0000" 2024-04-16T07:44:09.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.234 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.234 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.234 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.018941+0000" 2024-04-16T07:44:09.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.234 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.089631+0000" 2024-04-16T07:44:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:44:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:44:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 308, 2024-04-16T07:44:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:44:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:44:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.023589+0000" 2024-04-16T07:44:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:44:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.023608+0000" 2024-04-16T07:44:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:44:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.024754+0000" 2024-04-16T07:44:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:44:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:44:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.025137+0000" 2024-04-16T07:44:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:44:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.029230+0000" 2024-04-16T07:44:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:44:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:44:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:44:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:44:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:44:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:44:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:44:09.105301+0000" 2024-04-16T07:44:09.238 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.238 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.238 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:44:09.238 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-16T07:44:09.238 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-16T07:44:09.400 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 71 16 5440 5424 4.77409 12.75 0.0622278 0.209312 2024-04-16T07:44:10.400 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 72 16 5651 5635 4.89092 13.1875 0.074592 0.204325 2024-04-16T07:44:10.533 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:44:10.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:10.708+0000 7f9a7c06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:10.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:10.717+0000 7f9a7c06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:11.400 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 73 16 5848 5832 4.99257 12.3125 0.0608506 0.200158 2024-04-16T07:44:12.400 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 74 16 6041 6025 5.08809 12.0625 0.120717 0.196386 2024-04-16T07:44:13.112 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:44:13.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:13.284+0000 7fe053154640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:13.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:13.294+0000 7fe053154640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:13.400 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 75 16 6246 6230 5.19106 12.8125 0.0627475 0.192526 2024-04-16T07:44:14.240 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 1] 2024-04-16T07:44:14.240 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:44:14.241 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:44:14.401 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 76 16 6432 6416 5.2757 11.625 0.0975869 0.189454 2024-04-16T07:44:14.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:14.409+0000 7feed4aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:14.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:14.421+0000 7feed4aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:14.705 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:44:14.706 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-16T07:44:14.726 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-04-16T07:44:13.236168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":359989792,"num_objects":5486,"num_object_clones":0,"num_object_copies":16458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":5621,"num_write_kb":352870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":851968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5807,"ondisk_log_size":5807,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":213552,"kb_used_data":67344,"kb_used_omap":0,"kb_used_meta":146176,"kb_avail":377273808,"statfs":{"total":386547056640,"available":386328379392,"internally_reserved":0,"allocated":68960256,"data_stored":1090116348,"data_compressed":51755062,"data_compressed_allocated":68128768,"data_compressed_original":1089814528,"omap_allocated":0,"internal_metadata":149684224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":43778048,"num_objects":668,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":668,"num_write_kb":42752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":458752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.313819"},"pg_stats":[{"pgid":"3.f","version":"23'322","reported_seq":360,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:09.819675+0000","last_change":"2024-04-16T07:44:08.309230+0000","last_active":"2024-04-16T07:44:09.819675+0000","last_peered":"2024-04-16T07:44:09.819675+0000","last_clean":"2024-04-16T07:44:09.819675+0000","last_became_active":"2024-04-16T07:44:08.082516+0000","last_became_peered":"2024-04-16T07:44:08.082516+0000","last_unstale":"2024-04-16T07:44:09.819675+0000","last_undegraded":"2024-04-16T07:44:09.819675+0000","last_fullsized":"2024-04-16T07:44:09.819675+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21168128,"num_objects":323,"num_object_clones":0,"num_object_copies":969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":323,"num_write_kb":20672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"23'337","reported_seq":377,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:10.140629+0000","last_change":"2024-04-16T07:44:08.058565+0000","last_active":"2024-04-16T07:44:10.140629+0000","last_peered":"2024-04-16T07:44:10.140629+0000","last_clean":"2024-04-16T07:44:10.140629+0000","last_became_active":"2024-04-16T07:44:08.053424+0000","last_became_peered":"2024-04-16T07:44:08.053424+0000","last_unstale":"2024-04-16T07:44:10.140629+0000","last_undegraded":"2024-04-16T07:44:10.140629+0000","last_fullsized":"2024-04-16T07:44:10.140629+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22151168,"num_objects":338,"num_object_clones":0,"num_object_copies":1014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":338,"num_write_kb":21632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'362","reported_seq":394,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:11.572600+0000","last_change":"2024-04-16T07:44:08.134070+0000","last_active":"2024-04-16T07:44:11.572600+0000","last_peered":"2024-04-16T07:44:11.572600+0000","last_clean":"2024-04-16T07:44:11.572600+0000","last_became_active":"2024-04-16T07:44:08.128098+0000","last_became_peered":"2024-04-16T07:44:08.128098+0000","last_unstale":"2024-04-16T07:44:11.572600+0000","last_undegraded":"2024-04-16T07:44:11.572600+0000","last_fullsized":"2024-04-16T07:44:11.572600+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23789568,"num_objects":363,"num_object_clones":0,"num_object_copies":1089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":363,"num_write_kb":23232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'359","reported_seq":391,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:10.204364+0000","last_change":"2024-04-16T07:43:42.068753+0000","last_active":"2024-04-16T07:44:10.204364+0000","last_peered":"2024-04-16T07:44:10.204364+0000","last_clean":"2024-04-16T07:44:10.204364+0000","last_became_active":"2024-04-16T07:43:42.062710+0000","last_became_peered":"2024-04-16T07:43:42.062710+0000","last_unstale":"2024-04-16T07:44:10.204364+0000","last_undegraded":"2024-04-16T07:44:10.204364+0000","last_fullsized":"2024-04-16T07:44:10.204364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23592960,"num_objects":360,"num_object_clones":0,"num_object_copies":1080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":360,"num_write_kb":23040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"23'351","reported_seq":386,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:10.734611+0000","last_change":"2024-04-16T07:43:42.350463+0000","last_active":"2024-04-16T07:44:10.734611+0000","last_peered":"2024-04-16T07:44:10.734611+0000","last_clean":"2024-04-16T07:44:10.734611+0000","last_became_active":"2024-04-16T07:43:42.133631+0000","last_became_peered":"2024-04-16T07:43:42.133631+0000","last_unstale":"2024-04-16T07:44:10.734611+0000","last_undegraded":"2024-04-16T07:44:10.734611+0000","last_fullsized":"2024-04-16T07:44:10.734611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7208960,"num_objects":110,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":7040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"23'369","reported_seq":410,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:10.104295+0000","last_change":"2024-04-16T07:44:08.394287+0000","last_active":"2024-04-16T07:44:10.104295+0000","last_peered":"2024-04-16T07:44:10.104295+0000","last_clean":"2024-04-16T07:44:10.104295+0000","last_became_active":"2024-04-16T07:44:08.053918+0000","last_became_peered":"2024-04-16T07:44:08.053918+0000","last_unstale":"2024-04-16T07:44:10.104295+0000","last_undegraded":"2024-04-16T07:44:10.104295+0000","last_fullsized":"2024-04-16T07:44:10.104295+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24248320,"num_objects":370,"num_object_clones":0,"num_object_copies":1110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":370,"num_write_kb":23680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'333","reported_seq":365,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:10.178852+0000","last_change":"2024-04-16T07:44:08.071780+0000","last_active":"2024-04-16T07:44:10.178852+0000","last_peered":"2024-04-16T07:44:10.178852+0000","last_clean":"2024-04-16T07:44:10.178852+0000","last_became_active":"2024-04-16T07:44:08.062584+0000","last_became_peered":"2024-04-16T07:44:08.062584+0000","last_unstale":"2024-04-16T07:44:10.178852+0000","last_undegraded":"2024-04-16T07:44:10.178852+0000","last_fullsized":"2024-04-16T07:44:10.178852+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21889024,"num_objects":334,"num_object_clones":0,"num_object_copies":1002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":334,"num_write_kb":21376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"23'374","reported_seq":411,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:11.677722+0000","last_change":"2024-04-16T07:44:08.081912+0000","last_active":"2024-04-16T07:44:11.677722+0000","last_peered":"2024-04-16T07:44:11.677722+0000","last_clean":"2024-04-16T07:44:11.677722+0000","last_became_active":"2024-04-16T07:44:08.074439+0000","last_became_peered":"2024-04-16T07:44:08.074439+0000","last_unstale":"2024-04-16T07:44:11.677722+0000","last_undegraded":"2024-04-16T07:44:11.677722+0000","last_fullsized":"2024-04-16T07:44:11.677722+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24576000,"num_objects":375,"num_object_clones":0,"num_object_copies":1125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":375,"num_write_kb":24000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":157,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:07.954935+0000","last_change":"2024-04-16T07:43:42.109026+0000","last_active":"2024-04-16T07:44:07.954935+0000","last_peered":"2024-04-16T07:44:07.954935+0000","last_clean":"2024-04-16T07:44:07.954935+0000","last_became_active":"2024-04-16T07:43:42.097450+0000","last_became_peered":"2024-04-16T07:43:42.097450+0000","last_unstale":"2024-04-16T07:44:07.954935+0000","last_undegraded":"2024-04-16T07:44:07.954935+0000","last_fullsized":"2024-04-16T07:44:07.954935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_clean_scrub_stamp":"2024-04-16T07:42:42.229959+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:08.120395+0000","last_change":"2024-04-16T07:44:08.122308+0000","last_active":"2024-04-16T07:44:08.120395+0000","last_peered":"2024-04-16T07:44:08.120395+0000","last_clean":"2024-04-16T07:42:45.350220+0000","last_became_active":"2024-04-16T07:44:08.109936+0000","last_became_peered":"2024-04-16T07:44:08.109936+0000","last_unstale":"2024-04-16T07:44:08.120395+0000","last_undegraded":"2024-04-16T07:44:08.120395+0000","last_fullsized":"2024-04-16T07:44:08.120395+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"23'387","reported_seq":426,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:11.726205+0000","last_change":"2024-04-16T07:44:08.093755+0000","last_active":"2024-04-16T07:44:11.726205+0000","last_peered":"2024-04-16T07:44:11.726205+0000","last_clean":"2024-04-16T07:44:11.726205+0000","last_became_active":"2024-04-16T07:44:08.082731+0000","last_became_peered":"2024-04-16T07:44:08.082731+0000","last_unstale":"2024-04-16T07:44:11.726205+0000","last_undegraded":"2024-04-16T07:44:11.726205+0000","last_fullsized":"2024-04-16T07:44:11.726205+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":387,"log_dups_size":0,"ondisk_log_size":387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25427968,"num_objects":388,"num_object_clones":0,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":388,"num_write_kb":24832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:08.101357+0000","last_change":"2024-04-16T07:44:08.103194+0000","last_active":"2024-04-16T07:44:08.101357+0000","last_peered":"2024-04-16T07:44:08.101357+0000","last_clean":"2024-04-16T07:43:41.988602+0000","last_became_active":"2024-04-16T07:44:08.091632+0000","last_became_peered":"2024-04-16T07:44:08.091632+0000","last_unstale":"2024-04-16T07:44:08.101357+0000","last_undegraded":"2024-04-16T07:44:08.101357+0000","last_fullsized":"2024-04-16T07:44:08.101357+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"23'411","reported_seq":448,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:10.719572+0000","last_change":"2024-04-16T07:44:08.281808+0000","last_active":"2024-04-16T07:44:10.719572+0000","last_peered":"2024-04-16T07:44:10.719572+0000","last_clean":"2024-04-16T07:44:10.719572+0000","last_became_active":"2024-04-16T07:44:08.097957+0000","last_became_peered":"2024-04-16T07:44:08.097957+0000","last_unstale":"2024-04-16T07:44:10.719572+0000","last_undegraded":"2024-04-16T07:44:10.719572+0000","last_fullsized":"2024-04-16T07:44:10.719572+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":411,"log_dups_size":0,"ondisk_log_size":411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27000832,"num_objects":412,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":412,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:08.094035+0000","last_change":"2024-04-16T07:44:08.095970+0000","last_active":"2024-04-16T07:44:08.094035+0000","last_peered":"2024-04-16T07:44:08.094035+0000","last_clean":"2024-04-16T07:43:42.122878+0000","last_became_active":"2024-04-16T07:44:08.089191+0000","last_became_peered":"2024-04-16T07:44:08.089191+0000","last_unstale":"2024-04-16T07:44:08.094035+0000","last_undegraded":"2024-04-16T07:44:08.094035+0000","last_fullsized":"2024-04-16T07:44:08.094035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"23'370","reported_seq":407,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:10.278023+0000","last_change":"2024-04-16T07:44:08.189471+0000","last_active":"2024-04-16T07:44:10.278023+0000","last_peered":"2024-04-16T07:44:10.278023+0000","last_clean":"2024-04-16T07:44:10.278023+0000","last_became_active":"2024-04-16T07:44:08.059533+0000","last_became_peered":"2024-04-16T07:44:08.059533+0000","last_unstale":"2024-04-16T07:44:10.278023+0000","last_undegraded":"2024-04-16T07:44:10.278023+0000","last_fullsized":"2024-04-16T07:44:10.278023+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24313856,"num_objects":371,"num_object_clones":0,"num_object_copies":1113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":371,"num_write_kb":23744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:08.051590+0000","last_change":"2024-04-16T07:44:08.053706+0000","last_active":"2024-04-16T07:44:08.051590+0000","last_peered":"2024-04-16T07:44:08.051590+0000","last_clean":"2024-04-16T07:43:41.966913+0000","last_became_active":"2024-04-16T07:44:08.047032+0000","last_became_peered":"2024-04-16T07:44:08.047032+0000","last_unstale":"2024-04-16T07:44:08.051590+0000","last_undegraded":"2024-04-16T07:44:08.051590+0000","last_fullsized":"2024-04-16T07:44:08.051590+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"23'320","reported_seq":352,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:10.171816+0000","last_change":"2024-04-16T07:43:42.098380+0000","last_active":"2024-04-16T07:44:10.171816+0000","last_peered":"2024-04-16T07:44:10.171816+0000","last_clean":"2024-04-16T07:44:10.171816+0000","last_became_active":"2024-04-16T07:43:42.082881+0000","last_became_peered":"2024-04-16T07:43:42.082881+0000","last_unstale":"2024-04-16T07:44:10.171816+0000","last_undegraded":"2024-04-16T07:44:10.171816+0000","last_fullsized":"2024-04-16T07:44:10.171816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21037056,"num_objects":321,"num_object_clones":0,"num_object_copies":963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":321,"num_write_kb":20544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"23'359","reported_seq":391,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:11.761234+0000","last_change":"2024-04-16T07:44:08.117094+0000","last_active":"2024-04-16T07:44:11.761234+0000","last_peered":"2024-04-16T07:44:11.761234+0000","last_clean":"2024-04-16T07:44:11.761234+0000","last_became_active":"2024-04-16T07:44:08.102797+0000","last_became_peered":"2024-04-16T07:44:08.102797+0000","last_unstale":"2024-04-16T07:44:11.761234+0000","last_undegraded":"2024-04-16T07:44:11.761234+0000","last_fullsized":"2024-04-16T07:44:11.761234+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23592960,"num_objects":360,"num_object_clones":0,"num_object_copies":1080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":360,"num_write_kb":23040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:07.958218+0000","last_change":"2024-04-16T07:43:42.140887+0000","last_active":"2024-04-16T07:44:07.958218+0000","last_peered":"2024-04-16T07:44:07.958218+0000","last_clean":"2024-04-16T07:44:07.958218+0000","last_became_active":"2024-04-16T07:43:42.131062+0000","last_became_peered":"2024-04-16T07:43:42.131062+0000","last_unstale":"2024-04-16T07:44:07.958218+0000","last_undegraded":"2024-04-16T07:44:07.958218+0000","last_fullsized":"2024-04-16T07:44:07.958218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"23'345","reported_seq":382,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:10.120446+0000","last_change":"2024-04-16T07:43:42.299073+0000","last_active":"2024-04-16T07:44:10.120446+0000","last_peered":"2024-04-16T07:44:10.120446+0000","last_clean":"2024-04-16T07:44:10.120446+0000","last_became_active":"2024-04-16T07:43:42.114312+0000","last_became_peered":"2024-04-16T07:43:42.114312+0000","last_unstale":"2024-04-16T07:44:10.120446+0000","last_undegraded":"2024-04-16T07:44:10.120446+0000","last_fullsized":"2024-04-16T07:44:10.120446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22675456,"num_objects":346,"num_object_clones":0,"num_object_copies":1038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":346,"num_write_kb":22144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:07.978524+0000","last_change":"2024-04-16T07:43:42.120885+0000","last_active":"2024-04-16T07:44:07.978524+0000","last_peered":"2024-04-16T07:44:07.978524+0000","last_clean":"2024-04-16T07:44:07.978524+0000","last_became_active":"2024-04-16T07:43:42.114801+0000","last_became_peered":"2024-04-16T07:43:42.114801+0000","last_unstale":"2024-04-16T07:44:07.978524+0000","last_undegraded":"2024-04-16T07:44:07.978524+0000","last_fullsized":"2024-04-16T07:44:07.978524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"23'380","reported_seq":417,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:10.690382+0000","last_change":"2024-04-16T07:44:08.202317+0000","last_active":"2024-04-16T07:44:10.690382+0000","last_peered":"2024-04-16T07:44:10.690382+0000","last_clean":"2024-04-16T07:44:10.690382+0000","last_became_active":"2024-04-16T07:44:08.051619+0000","last_became_peered":"2024-04-16T07:44:08.051619+0000","last_unstale":"2024-04-16T07:44:10.690382+0000","last_undegraded":"2024-04-16T07:44:10.690382+0000","last_fullsized":"2024-04-16T07:44:10.690382+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":380,"log_dups_size":0,"ondisk_log_size":380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24969216,"num_objects":381,"num_object_clones":0,"num_object_copies":1143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":381,"num_write_kb":24384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:07.946387+0000","last_change":"2024-04-16T07:43:42.077187+0000","last_active":"2024-04-16T07:44:07.946387+0000","last_peered":"2024-04-16T07:44:07.946387+0000","last_clean":"2024-04-16T07:44:07.946387+0000","last_became_active":"2024-04-16T07:43:42.070018+0000","last_became_peered":"2024-04-16T07:43:42.070018+0000","last_unstale":"2024-04-16T07:44:07.946387+0000","last_undegraded":"2024-04-16T07:44:07.946387+0000","last_fullsized":"2024-04-16T07:44:07.946387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"23'331","reported_seq":364,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:10.113174+0000","last_change":"2024-04-16T07:44:08.058741+0000","last_active":"2024-04-16T07:44:10.113174+0000","last_peered":"2024-04-16T07:44:10.113174+0000","last_clean":"2024-04-16T07:44:10.113174+0000","last_became_active":"2024-04-16T07:44:08.053159+0000","last_became_peered":"2024-04-16T07:44:08.053159+0000","last_unstale":"2024-04-16T07:44:10.113174+0000","last_undegraded":"2024-04-16T07:44:10.113174+0000","last_fullsized":"2024-04-16T07:44:10.113174+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21757952,"num_objects":332,"num_object_clones":0,"num_object_copies":996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":332,"num_write_kb":21248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:08.120070+0000","last_change":"2024-04-16T07:44:08.122061+0000","last_active":"2024-04-16T07:44:08.120070+0000","last_peered":"2024-04-16T07:44:08.120070+0000","last_clean":"2024-04-16T07:42:45.376121+0000","last_became_active":"2024-04-16T07:44:08.114179+0000","last_became_peered":"2024-04-16T07:44:08.114179+0000","last_unstale":"2024-04-16T07:44:08.120070+0000","last_undegraded":"2024-04-16T07:44:08.120070+0000","last_fullsized":"2024-04-16T07:44:08.120070+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":359399424,"num_objects":5484,"num_object_clones":0,"num_object_copies":16452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5484,"num_write_kb":350976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":851968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5710,"ondisk_log_size":5710,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51352,"kb_used_data":19664,"kb_used_omap":0,"kb_used_meta":31680,"kb_avail":94320488,"statfs":{"total":96636764160,"available":96584179712,"internally_reserved":0,"allocated":20135936,"data_stored":318764871,"data_compressed":15134623,"data_compressed_allocated":19922944,"data_compressed_original":318685184,"omap_allocated":0,"internal_metadata":32440320},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":58336,"kb_used_data":14872,"kb_used_omap":0,"kb_used_meta":43456,"kb_avail":94313504,"statfs":{"total":96636764160,"available":96577028096,"internally_reserved":0,"allocated":15228928,"data_stored":240252743,"data_compressed":11406680,"data_compressed_allocated":15015936,"data_compressed_original":240173056,"omap_allocated":0,"internal_metadata":44498944},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":45148,"kb_used_data":17556,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94326692,"statfs":{"total":96636764160,"available":96590532608,"internally_reserved":0,"allocated":17977344,"data_stored":284620071,"data_compressed":13511450,"data_compressed_allocated":17784832,"data_compressed_original":284557312,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":58716,"kb_used_data":15252,"kb_used_omap":0,"kb_used_meta":43456,"kb_avail":94313124,"statfs":{"total":96636764160,"available":96576638976,"internally_reserved":0,"allocated":15618048,"data_stored":246478663,"data_compressed":11702309,"data_compressed_allocated":15405056,"data_compressed_original":246398976,"omap_allocated":0,"internal_metadata":44498944},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:44:15.401 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 77 16 6618 6602 5.35815 11.625 0.0480458 0.186502 2024-04-16T07:44:15.722 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:44:15.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:15.880+0000 7fca1759d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:15.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:15.889+0000 7fca1759d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:16.401 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 78 16 6799 6783 5.43447 11.3125 0.0360711 0.183856 2024-04-16T07:44:17.401 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 79 16 7001 6985 5.52547 12.625 0.101769 0.180909 2024-04-16T07:44:18.253 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:44:18.401 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:2024-04-16T07:44:18.402785+0000 min lat: 0.0168854 max lat: 25.1577 avg lat: 0.178233 2024-04-16T07:44:18.401 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:44:18.401 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 80 16 7193 7177 5.60639 12 0.0704082 0.178233 2024-04-16T07:44:18.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:18.412+0000 7fdb35b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:18.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:18.422+0000 7fdb35b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:19.401 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 81 16 7386 7370 5.68608 12.0625 0.120531 0.175787 2024-04-16T07:44:19.728 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 1] 2024-04-16T07:44:19.728 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:44:19.728 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-16T07:44:19.729 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-16T07:44:19.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:19.897+0000 7f4bd56ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:19.911+0000 7f4bd56ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-04-16T07:44:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:config get 2024-04-16T07:44:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:config help 2024-04-16T07:44:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:config set ... 2024-04-16T07:44:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:config show 2024-04-16T07:44:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:deep_scrub 2024-04-16T07:44:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_ops 2024-04-16T07:44:20.002 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_slow_ops 2024-04-16T07:44:20.002 INFO:teuthology.orchestra.run.smithi031.stderr:dump_metrics [] 2024-04-16T07:44:20.002 INFO:teuthology.orchestra.run.smithi031.stderr:dump_ops_in_flight 2024-04-16T07:44:20.002 INFO:teuthology.orchestra.run.smithi031.stderr:dump_pgstate_history 2024-04-16T07:44:20.002 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-04-16T07:44:20.004 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:44:20.004 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:44:20.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-16T07:44:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:20.166+0000 7fa86939d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:20.186+0000 7fa86939d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:20.306 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-04-16T07:44:20.307 INFO:teuthology.orchestra.run.smithi031.stderr:config get 2024-04-16T07:44:20.307 INFO:teuthology.orchestra.run.smithi031.stderr:config help 2024-04-16T07:44:20.307 INFO:teuthology.orchestra.run.smithi031.stderr:config set ... 2024-04-16T07:44:20.307 INFO:teuthology.orchestra.run.smithi031.stderr:config show 2024-04-16T07:44:20.307 INFO:teuthology.orchestra.run.smithi031.stderr:deep_scrub 2024-04-16T07:44:20.307 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_ops 2024-04-16T07:44:20.307 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_slow_ops 2024-04-16T07:44:20.308 INFO:teuthology.orchestra.run.smithi031.stderr:dump_metrics [] 2024-04-16T07:44:20.308 INFO:teuthology.orchestra.run.smithi031.stderr:dump_ops_in_flight 2024-04-16T07:44:20.308 INFO:teuthology.orchestra.run.smithi031.stderr:dump_pgstate_history 2024-04-16T07:44:20.308 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-04-16T07:44:20.310 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:44:20.311 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:44:20.311 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-16T07:44:20.401 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 82 16 7585 7569 5.7684 12.4375 0.0628277 0.173242 2024-04-16T07:44:20.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:20.469+0000 7f359079d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:20.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:20.483+0000 7f359079d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:20.504 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-04-16T07:44:20.504 INFO:teuthology.orchestra.run.smithi031.stderr:config get 2024-04-16T07:44:20.505 INFO:teuthology.orchestra.run.smithi031.stderr:config help 2024-04-16T07:44:20.505 INFO:teuthology.orchestra.run.smithi031.stderr:config set ... 2024-04-16T07:44:20.505 INFO:teuthology.orchestra.run.smithi031.stderr:config show 2024-04-16T07:44:20.505 INFO:teuthology.orchestra.run.smithi031.stderr:deep_scrub 2024-04-16T07:44:20.505 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_ops 2024-04-16T07:44:20.505 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_slow_ops 2024-04-16T07:44:20.505 INFO:teuthology.orchestra.run.smithi031.stderr:dump_metrics [] 2024-04-16T07:44:20.505 INFO:teuthology.orchestra.run.smithi031.stderr:dump_ops_in_flight 2024-04-16T07:44:20.505 INFO:teuthology.orchestra.run.smithi031.stderr:dump_pgstate_history 2024-04-16T07:44:20.505 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-04-16T07:44:20.509 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:44:20.510 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:44:20.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-16T07:44:20.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:20.678+0000 7f9bb435d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:20.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:20.699+0000 7f9bb435d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:20.758 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-04-16T07:44:20.758 INFO:teuthology.orchestra.run.smithi031.stderr:config get 2024-04-16T07:44:20.758 INFO:teuthology.orchestra.run.smithi031.stderr:config help 2024-04-16T07:44:20.758 INFO:teuthology.orchestra.run.smithi031.stderr:config set ... 2024-04-16T07:44:20.758 INFO:teuthology.orchestra.run.smithi031.stderr:config show 2024-04-16T07:44:20.758 INFO:teuthology.orchestra.run.smithi031.stderr:deep_scrub 2024-04-16T07:44:20.758 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_ops 2024-04-16T07:44:20.758 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_slow_ops 2024-04-16T07:44:20.758 INFO:teuthology.orchestra.run.smithi031.stderr:dump_metrics [] 2024-04-16T07:44:20.759 INFO:teuthology.orchestra.run.smithi031.stderr:dump_ops_in_flight 2024-04-16T07:44:20.759 INFO:teuthology.orchestra.run.smithi031.stderr:dump_pgstate_history 2024-04-16T07:44:20.759 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-04-16T07:44:20.763 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:44:20.764 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:44:20.767 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:44:20.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:20.922+0000 7f5d692ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:20.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:20.930+0000 7f5d692ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:21.401 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 83 16 7777 7761 5.84346 12 0.130167 0.171026 2024-04-16T07:44:22.401 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 84 16 7971 7955 5.91823 12.125 0.138757 0.168853 2024-04-16T07:44:23.391 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:44:23.401 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 85 16 8163 8147 5.98976 12 0.0692496 0.166852 2024-04-16T07:44:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:23.554+0000 7f95ced9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:23.564+0000 7f95ced9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:24.401 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 86 16 8343 8327 6.05092 11.25 0.0594347 0.165134 2024-04-16T07:44:25.402 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 87 16 8543 8527 6.12503 12.5 0.121627 0.163173 2024-04-16T07:44:25.765 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 1] 2024-04-16T07:44:25.765 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:44:25.765 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 2, 3, 1] 2024-04-16T07:44:25.766 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-16T07:44:25.766 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:44:25.840 INFO:tasks.ceph.osd.3:Stopped 2024-04-16T07:44:26.006 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:44:26.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:26.120+0000 7fd7a319d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:26.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:26.126+0000 7fd7a319d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:26.402 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 88 16 8626 8610 6.11437 5.1875 0.0565001 0.162372 2024-04-16T07:44:27.402 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 89 16 8626 8610 6.04567 0 - 0.162372 2024-04-16T07:44:28.402 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 90 16 8626 8610 5.97849 0 - 0.162372 2024-04-16T07:44:29.402 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 91 16 8626 8610 5.91279 0 - 0.162372 2024-04-16T07:44:30.402 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 92 16 8626 8610 5.84852 0 - 0.162372 2024-04-16T07:44:30.841 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [0, 2, 1] 2024-04-16T07:44:30.841 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:44:30.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:44:30.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:30.967+0000 7f78cd86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:30.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:30.976+0000 7f78cd86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:31.176 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:44:31.177 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-16T07:44:31.189 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-04-16T07:44:29.239896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":529531424,"num_objects":8073,"num_object_clones":0,"num_object_copies":24219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":8208,"num_write_kb":518438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":851968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8394,"ondisk_log_size":8394,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":236744,"kb_used_data":98852,"kb_used_omap":0,"kb_used_meta":137856,"kb_avail":377250616,"statfs":{"total":386547056640,"available":386304630784,"internally_reserved":0,"allocated":101224448,"data_stored":1606343420,"data_compressed":76268286,"data_compressed_allocated":100392960,"data_compressed_original":1606041600,"omap_allocated":0,"internal_metadata":141164544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":106627072,"num_objects":1627,"num_object_clones":0,"num_object_copies":4881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1627,"num_write_kb":104128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002907"},"pg_stats":[{"pgid":"3.f","version":"23'509","reported_seq":547,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:25.197136+0000","last_change":"2024-04-16T07:44:08.309230+0000","last_active":"2024-04-16T07:44:25.197136+0000","last_peered":"2024-04-16T07:44:25.197136+0000","last_clean":"2024-04-16T07:44:25.197136+0000","last_became_active":"2024-04-16T07:44:08.082516+0000","last_became_peered":"2024-04-16T07:44:08.082516+0000","last_unstale":"2024-04-16T07:44:25.197136+0000","last_undegraded":"2024-04-16T07:44:25.197136+0000","last_fullsized":"2024-04-16T07:44:25.197136+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33423360,"num_objects":510,"num_object_clones":0,"num_object_copies":1530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":510,"num_write_kb":32640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"23'499","reported_seq":539,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:25.043369+0000","last_change":"2024-04-16T07:44:08.058565+0000","last_active":"2024-04-16T07:44:25.043369+0000","last_peered":"2024-04-16T07:44:25.043369+0000","last_clean":"2024-04-16T07:44:25.043369+0000","last_became_active":"2024-04-16T07:44:08.053424+0000","last_became_peered":"2024-04-16T07:44:08.053424+0000","last_unstale":"2024-04-16T07:44:25.043369+0000","last_undegraded":"2024-04-16T07:44:25.043369+0000","last_fullsized":"2024-04-16T07:44:25.043369+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":500,"num_object_clones":0,"num_object_copies":1500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":500,"num_write_kb":32000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'516","reported_seq":548,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:25.787182+0000","last_change":"2024-04-16T07:44:08.134070+0000","last_active":"2024-04-16T07:44:25.787182+0000","last_peered":"2024-04-16T07:44:25.787182+0000","last_clean":"2024-04-16T07:44:25.787182+0000","last_became_active":"2024-04-16T07:44:08.128098+0000","last_became_peered":"2024-04-16T07:44:08.128098+0000","last_unstale":"2024-04-16T07:44:25.787182+0000","last_undegraded":"2024-04-16T07:44:25.787182+0000","last_fullsized":"2024-04-16T07:44:25.787182+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":516,"log_dups_size":0,"ondisk_log_size":516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33882112,"num_objects":517,"num_object_clones":0,"num_object_copies":1551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":517,"num_write_kb":33088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'548","reported_seq":580,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:25.212124+0000","last_change":"2024-04-16T07:43:42.068753+0000","last_active":"2024-04-16T07:44:25.212124+0000","last_peered":"2024-04-16T07:44:25.212124+0000","last_clean":"2024-04-16T07:44:25.212124+0000","last_became_active":"2024-04-16T07:43:42.062710+0000","last_became_peered":"2024-04-16T07:43:42.062710+0000","last_unstale":"2024-04-16T07:44:25.212124+0000","last_undegraded":"2024-04-16T07:44:25.212124+0000","last_fullsized":"2024-04-16T07:44:25.212124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35979264,"num_objects":549,"num_object_clones":0,"num_object_copies":1647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":549,"num_write_kb":35136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"23'478","reported_seq":513,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:20.655835+0000","last_change":"2024-04-16T07:43:42.350463+0000","last_active":"2024-04-16T07:44:20.655835+0000","last_peered":"2024-04-16T07:44:20.655835+0000","last_clean":"2024-04-16T07:44:20.655835+0000","last_became_active":"2024-04-16T07:43:42.133631+0000","last_became_peered":"2024-04-16T07:43:42.133631+0000","last_unstale":"2024-04-16T07:44:20.655835+0000","last_undegraded":"2024-04-16T07:44:20.655835+0000","last_fullsized":"2024-04-16T07:44:20.655835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":478,"log_dups_size":0,"ondisk_log_size":478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15532032,"num_objects":237,"num_object_clones":0,"num_object_copies":711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":237,"num_write_kb":15168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"23'542","reported_seq":583,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:25.192251+0000","last_change":"2024-04-16T07:44:08.394287+0000","last_active":"2024-04-16T07:44:25.192251+0000","last_peered":"2024-04-16T07:44:25.192251+0000","last_clean":"2024-04-16T07:44:25.192251+0000","last_became_active":"2024-04-16T07:44:08.053918+0000","last_became_peered":"2024-04-16T07:44:08.053918+0000","last_unstale":"2024-04-16T07:44:25.192251+0000","last_undegraded":"2024-04-16T07:44:25.192251+0000","last_fullsized":"2024-04-16T07:44:25.192251+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":542,"log_dups_size":0,"ondisk_log_size":542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35586048,"num_objects":543,"num_object_clones":0,"num_object_copies":1629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":543,"num_write_kb":34752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'511","reported_seq":543,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:25.040625+0000","last_change":"2024-04-16T07:44:08.071780+0000","last_active":"2024-04-16T07:44:25.040625+0000","last_peered":"2024-04-16T07:44:25.040625+0000","last_clean":"2024-04-16T07:44:25.040625+0000","last_became_active":"2024-04-16T07:44:08.062584+0000","last_became_peered":"2024-04-16T07:44:08.062584+0000","last_unstale":"2024-04-16T07:44:25.040625+0000","last_undegraded":"2024-04-16T07:44:25.040625+0000","last_fullsized":"2024-04-16T07:44:25.040625+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":512,"num_object_clones":0,"num_object_copies":1536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":512,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"23'528","reported_seq":565,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:25.623187+0000","last_change":"2024-04-16T07:44:08.081912+0000","last_active":"2024-04-16T07:44:25.623187+0000","last_peered":"2024-04-16T07:44:25.623187+0000","last_clean":"2024-04-16T07:44:25.623187+0000","last_became_active":"2024-04-16T07:44:08.074439+0000","last_became_peered":"2024-04-16T07:44:08.074439+0000","last_unstale":"2024-04-16T07:44:25.623187+0000","last_undegraded":"2024-04-16T07:44:25.623187+0000","last_fullsized":"2024-04-16T07:44:25.623187+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":528,"log_dups_size":0,"ondisk_log_size":528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34668544,"num_objects":529,"num_object_clones":0,"num_object_copies":1587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":529,"num_write_kb":33856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":157,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:07.954935+0000","last_change":"2024-04-16T07:43:42.109026+0000","last_active":"2024-04-16T07:44:07.954935+0000","last_peered":"2024-04-16T07:44:07.954935+0000","last_clean":"2024-04-16T07:44:07.954935+0000","last_became_active":"2024-04-16T07:43:42.097450+0000","last_became_peered":"2024-04-16T07:43:42.097450+0000","last_unstale":"2024-04-16T07:44:07.954935+0000","last_undegraded":"2024-04-16T07:44:07.954935+0000","last_fullsized":"2024-04-16T07:44:07.954935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_clean_scrub_stamp":"2024-04-16T07:42:42.229959+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:08.120395+0000","last_change":"2024-04-16T07:44:08.122308+0000","last_active":"2024-04-16T07:44:08.120395+0000","last_peered":"2024-04-16T07:44:08.120395+0000","last_clean":"2024-04-16T07:42:45.350220+0000","last_became_active":"2024-04-16T07:44:08.109936+0000","last_became_peered":"2024-04-16T07:44:08.109936+0000","last_unstale":"2024-04-16T07:44:08.120395+0000","last_undegraded":"2024-04-16T07:44:08.120395+0000","last_fullsized":"2024-04-16T07:44:08.120395+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"23'556","reported_seq":595,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:25.673897+0000","last_change":"2024-04-16T07:44:08.093755+0000","last_active":"2024-04-16T07:44:25.673897+0000","last_peered":"2024-04-16T07:44:25.673897+0000","last_clean":"2024-04-16T07:44:25.673897+0000","last_became_active":"2024-04-16T07:44:08.082731+0000","last_became_peered":"2024-04-16T07:44:08.082731+0000","last_unstale":"2024-04-16T07:44:25.673897+0000","last_undegraded":"2024-04-16T07:44:25.673897+0000","last_fullsized":"2024-04-16T07:44:25.673897+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36503552,"num_objects":557,"num_object_clones":0,"num_object_copies":1671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":557,"num_write_kb":35648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:08.101357+0000","last_change":"2024-04-16T07:44:08.103194+0000","last_active":"2024-04-16T07:44:08.101357+0000","last_peered":"2024-04-16T07:44:08.101357+0000","last_clean":"2024-04-16T07:43:41.988602+0000","last_became_active":"2024-04-16T07:44:08.091632+0000","last_became_peered":"2024-04-16T07:44:08.091632+0000","last_unstale":"2024-04-16T07:44:08.101357+0000","last_undegraded":"2024-04-16T07:44:08.101357+0000","last_fullsized":"2024-04-16T07:44:08.101357+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"23'526","reported_seq":563,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:20.615797+0000","last_change":"2024-04-16T07:44:08.281808+0000","last_active":"2024-04-16T07:44:20.615797+0000","last_peered":"2024-04-16T07:44:20.615797+0000","last_clean":"2024-04-16T07:44:20.615797+0000","last_became_active":"2024-04-16T07:44:08.097957+0000","last_became_peered":"2024-04-16T07:44:08.097957+0000","last_unstale":"2024-04-16T07:44:20.615797+0000","last_undegraded":"2024-04-16T07:44:20.615797+0000","last_fullsized":"2024-04-16T07:44:20.615797+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":526,"log_dups_size":0,"ondisk_log_size":526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34537472,"num_objects":527,"num_object_clones":0,"num_object_copies":1581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":527,"num_write_kb":33728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:08.094035+0000","last_change":"2024-04-16T07:44:08.095970+0000","last_active":"2024-04-16T07:44:08.094035+0000","last_peered":"2024-04-16T07:44:08.094035+0000","last_clean":"2024-04-16T07:43:42.122878+0000","last_became_active":"2024-04-16T07:44:08.089191+0000","last_became_peered":"2024-04-16T07:44:08.089191+0000","last_unstale":"2024-04-16T07:44:08.094035+0000","last_undegraded":"2024-04-16T07:44:08.094035+0000","last_fullsized":"2024-04-16T07:44:08.094035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"23'488","reported_seq":525,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:20.356291+0000","last_change":"2024-04-16T07:44:08.189471+0000","last_active":"2024-04-16T07:44:20.356291+0000","last_peered":"2024-04-16T07:44:20.356291+0000","last_clean":"2024-04-16T07:44:20.356291+0000","last_became_active":"2024-04-16T07:44:08.059533+0000","last_became_peered":"2024-04-16T07:44:08.059533+0000","last_unstale":"2024-04-16T07:44:20.356291+0000","last_undegraded":"2024-04-16T07:44:20.356291+0000","last_fullsized":"2024-04-16T07:44:20.356291+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32047104,"num_objects":489,"num_object_clones":0,"num_object_copies":1467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":489,"num_write_kb":31296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:08.051590+0000","last_change":"2024-04-16T07:44:08.053706+0000","last_active":"2024-04-16T07:44:08.051590+0000","last_peered":"2024-04-16T07:44:08.051590+0000","last_clean":"2024-04-16T07:43:41.966913+0000","last_became_active":"2024-04-16T07:44:08.047032+0000","last_became_peered":"2024-04-16T07:44:08.047032+0000","last_unstale":"2024-04-16T07:44:08.051590+0000","last_undegraded":"2024-04-16T07:44:08.051590+0000","last_fullsized":"2024-04-16T07:44:08.051590+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"23'495","reported_seq":527,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:25.053544+0000","last_change":"2024-04-16T07:43:42.098380+0000","last_active":"2024-04-16T07:44:25.053544+0000","last_peered":"2024-04-16T07:44:25.053544+0000","last_clean":"2024-04-16T07:44:25.053544+0000","last_became_active":"2024-04-16T07:43:42.082881+0000","last_became_peered":"2024-04-16T07:43:42.082881+0000","last_unstale":"2024-04-16T07:44:25.053544+0000","last_undegraded":"2024-04-16T07:44:25.053544+0000","last_fullsized":"2024-04-16T07:44:25.053544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32505856,"num_objects":496,"num_object_clones":0,"num_object_copies":1488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":496,"num_write_kb":31744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"23'540","reported_seq":572,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:25.753259+0000","last_change":"2024-04-16T07:44:08.117094+0000","last_active":"2024-04-16T07:44:25.753259+0000","last_peered":"2024-04-16T07:44:25.753259+0000","last_clean":"2024-04-16T07:44:25.753259+0000","last_became_active":"2024-04-16T07:44:08.102797+0000","last_became_peered":"2024-04-16T07:44:08.102797+0000","last_unstale":"2024-04-16T07:44:25.753259+0000","last_undegraded":"2024-04-16T07:44:25.753259+0000","last_fullsized":"2024-04-16T07:44:25.753259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35454976,"num_objects":541,"num_object_clones":0,"num_object_copies":1623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":541,"num_write_kb":34624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:07.958218+0000","last_change":"2024-04-16T07:43:42.140887+0000","last_active":"2024-04-16T07:44:07.958218+0000","last_peered":"2024-04-16T07:44:07.958218+0000","last_clean":"2024-04-16T07:44:07.958218+0000","last_became_active":"2024-04-16T07:43:42.131062+0000","last_became_peered":"2024-04-16T07:43:42.131062+0000","last_unstale":"2024-04-16T07:44:07.958218+0000","last_undegraded":"2024-04-16T07:44:07.958218+0000","last_fullsized":"2024-04-16T07:44:07.958218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"23'542","reported_seq":579,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:25.229158+0000","last_change":"2024-04-16T07:43:42.299073+0000","last_active":"2024-04-16T07:44:25.229158+0000","last_peered":"2024-04-16T07:44:25.229158+0000","last_clean":"2024-04-16T07:44:25.229158+0000","last_became_active":"2024-04-16T07:43:42.114312+0000","last_became_peered":"2024-04-16T07:43:42.114312+0000","last_unstale":"2024-04-16T07:44:25.229158+0000","last_undegraded":"2024-04-16T07:44:25.229158+0000","last_fullsized":"2024-04-16T07:44:25.229158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":542,"log_dups_size":0,"ondisk_log_size":542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35586048,"num_objects":543,"num_object_clones":0,"num_object_copies":1629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":543,"num_write_kb":34752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:07.978524+0000","last_change":"2024-04-16T07:43:42.120885+0000","last_active":"2024-04-16T07:44:07.978524+0000","last_peered":"2024-04-16T07:44:07.978524+0000","last_clean":"2024-04-16T07:44:07.978524+0000","last_became_active":"2024-04-16T07:43:42.114801+0000","last_became_peered":"2024-04-16T07:43:42.114801+0000","last_unstale":"2024-04-16T07:44:07.978524+0000","last_undegraded":"2024-04-16T07:44:07.978524+0000","last_fullsized":"2024-04-16T07:44:07.978524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"23'507","reported_seq":544,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:20.701961+0000","last_change":"2024-04-16T07:44:08.202317+0000","last_active":"2024-04-16T07:44:20.701961+0000","last_peered":"2024-04-16T07:44:20.701961+0000","last_clean":"2024-04-16T07:44:20.701961+0000","last_became_active":"2024-04-16T07:44:08.051619+0000","last_became_peered":"2024-04-16T07:44:08.051619+0000","last_unstale":"2024-04-16T07:44:20.701961+0000","last_undegraded":"2024-04-16T07:44:20.701961+0000","last_fullsized":"2024-04-16T07:44:20.701961+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33292288,"num_objects":508,"num_object_clones":0,"num_object_copies":1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":508,"num_write_kb":32512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:07.946387+0000","last_change":"2024-04-16T07:43:42.077187+0000","last_active":"2024-04-16T07:44:07.946387+0000","last_peered":"2024-04-16T07:44:07.946387+0000","last_clean":"2024-04-16T07:44:07.946387+0000","last_became_active":"2024-04-16T07:43:42.070018+0000","last_became_peered":"2024-04-16T07:43:42.070018+0000","last_unstale":"2024-04-16T07:44:07.946387+0000","last_undegraded":"2024-04-16T07:44:07.946387+0000","last_fullsized":"2024-04-16T07:44:07.946387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"23'512","reported_seq":545,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:24.815767+0000","last_change":"2024-04-16T07:44:08.058741+0000","last_active":"2024-04-16T07:44:24.815767+0000","last_peered":"2024-04-16T07:44:24.815767+0000","last_clean":"2024-04-16T07:44:24.815767+0000","last_became_active":"2024-04-16T07:44:08.053159+0000","last_became_peered":"2024-04-16T07:44:08.053159+0000","last_unstale":"2024-04-16T07:44:24.815767+0000","last_undegraded":"2024-04-16T07:44:24.815767+0000","last_fullsized":"2024-04-16T07:44:24.815767+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33619968,"num_objects":513,"num_object_clones":0,"num_object_copies":1539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":513,"num_write_kb":32832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:44:08.120070+0000","last_change":"2024-04-16T07:44:08.122061+0000","last_active":"2024-04-16T07:44:08.120070+0000","last_peered":"2024-04-16T07:44:08.120070+0000","last_clean":"2024-04-16T07:42:45.376121+0000","last_became_active":"2024-04-16T07:44:08.114179+0000","last_became_peered":"2024-04-16T07:44:08.114179+0000","last_unstale":"2024-04-16T07:44:08.120070+0000","last_undegraded":"2024-04-16T07:44:08.120070+0000","last_fullsized":"2024-04-16T07:44:08.120070+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":528941056,"num_objects":8071,"num_object_clones":0,"num_object_copies":24213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8071,"num_write_kb":516544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":851968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8297,"ondisk_log_size":8297,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378663,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62380,"kb_used_data":26528,"kb_used_omap":0,"kb_used_meta":35840,"kb_avail":94309460,"statfs":{"total":96636764160,"available":96572887040,"internally_reserved":0,"allocated":27164672,"data_stored":431224647,"data_compressed":20474815,"data_compressed_allocated":26951680,"data_compressed_original":431144960,"omap_allocated":0,"internal_metadata":36700160},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":53868,"kb_used_data":22884,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94317972,"statfs":{"total":96636764160,"available":96581603328,"internally_reserved":0,"allocated":23433216,"data_stored":371521351,"data_compressed":17640016,"data_compressed_allocated":23220224,"data_compressed_original":371441664,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280531,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":61972,"kb_used_data":26060,"kb_used_omap":0,"kb_used_meta":35904,"kb_avail":94309868,"statfs":{"total":96636764160,"available":96573304832,"internally_reserved":0,"allocated":26685440,"data_stored":423949607,"data_compressed":20127562,"data_compressed_allocated":26492928,"data_compressed_original":423886848,"omap_allocated":0,"internal_metadata":36765696},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705768,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":58524,"kb_used_data":23380,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94313316,"statfs":{"total":96636764160,"available":96576835584,"internally_reserved":0,"allocated":23941120,"data_stored":379647815,"data_compressed":18025893,"data_compressed_allocated":23728128,"data_compressed_original":379568128,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:44:31.403 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 93 16 8626 8610 5.78563 0 - 0.162372 2024-04-16T07:44:32.403 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 94 16 8626 8610 5.72408 0 - 0.162372 2024-04-16T07:44:33.403 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 95 16 8626 8610 5.66382 0 - 0.162372 2024-04-16T07:44:34.403 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 96 16 8626 8610 5.60482 0 - 0.162372 2024-04-16T07:44:35.403 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 97 16 8626 8610 5.54704 0 - 0.162372 2024-04-16T07:44:36.191 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [0, 2, 1] 2024-04-16T07:44:36.191 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:44:36.191 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 2, 1] 2024-04-16T07:44:36.192 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-16T07:44:36.192 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:44:36.264 INFO:tasks.ceph.osd.0:Stopped 2024-04-16T07:44:36.403 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 98 16 8626 8610 5.49043 0 - 0.162372 2024-04-16T07:44:37.404 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 99 16 8626 8610 5.43497 0 - 0.162372 2024-04-16T07:44:38.404 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:2024-04-16T07:44:38.405337+0000 min lat: 0.0168854 max lat: 25.1577 avg lat: 0.162372 2024-04-16T07:44:38.404 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:44:38.404 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 100 16 8626 8610 5.38062 0 - 0.162372 2024-04-16T07:44:39.404 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 101 16 8626 8610 5.32734 0 - 0.162372 2024-04-16T07:44:40.404 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 102 16 8626 8610 5.27511 0 - 0.162372 2024-04-16T07:44:41.265 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3, 0] live_osds: [2, 1] 2024-04-16T07:44:41.265 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:44:41.266 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-16T07:44:41.266 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-16T07:44:41.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:41.392+0000 7f272ff1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:41.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:41.401+0000 7f272ff1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:41.404 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 103 16 8626 8610 5.2239 0 - 0.162372 2024-04-16T07:44:42.404 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 104 16 8626 8610 5.17367 0 - 0.162372 2024-04-16T07:44:43.404 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 105 16 8626 8610 5.12439 0 - 0.162372 2024-04-16T07:44:44.404 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 106 16 8626 8610 5.07605 0 - 0.162372 2024-04-16T07:44:45.405 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 107 16 8626 8610 5.02861 0 - 0.162372 2024-04-16T07:44:46.405 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 108 16 8626 8610 4.98205 0 - 0.162372 2024-04-16T07:44:47.405 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 109 16 8626 8610 4.93634 0 - 0.162372 2024-04-16T07:44:48.405 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 110 16 8626 8610 4.89147 0 - 0.162372 2024-04-16T07:44:49.405 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 111 16 8626 8610 4.8474 0 - 0.162372 2024-04-16T07:44:50.405 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 112 16 8626 8610 4.80412 0 - 0.162372 2024-04-16T07:44:51.405 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 113 16 8626 8610 4.7616 0 - 0.162372 2024-04-16T07:44:52.406 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 114 16 8626 8610 4.71983 0 - 0.162372 2024-04-16T07:44:53.406 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 115 16 8626 8610 4.67879 0 - 0.162372 2024-04-16T07:44:53.952 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:44:53.957 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:44:53.957 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:44:54.406 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 116 16 8626 8610 4.63845 0 - 0.162372 2024-04-16T07:44:55.406 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 117 16 8636 8620 4.60415 0.0215517 0.0196317 0.182567 2024-04-16T07:44:55.959 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:44:56.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:56.084+0000 7fad0f99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:56.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:44:56.090+0000 7fad0f99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:56.406 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 118 16 8636 8620 4.56513 0 - 0.182567 2024-04-16T07:44:57.406 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 119 16 8636 8620 4.52677 0 - 0.182567 2024-04-16T07:44:58.406 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:2024-04-16T07:44:58.408048+0000 min lat: 0.0165093 max lat: 29.3065 avg lat: 0.182567 2024-04-16T07:44:58.407 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:44:58.407 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 120 16 8636 8620 4.48904 0 - 0.182567 2024-04-16T07:44:59.406 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 121 16 8636 8620 4.45194 0 - 0.182567 2024-04-16T07:45:00.407 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 122 16 8636 8620 4.41545 0 - 0.182567 2024-04-16T07:45:01.407 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 123 16 8636 8620 4.37955 0 - 0.182567 2024-04-16T07:45:02.407 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 124 16 8636 8620 4.34423 0 - 0.182567 2024-04-16T07:45:03.407 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 125 16 8636 8620 4.30948 0 - 0.182567 2024-04-16T07:45:04.407 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 126 16 8636 8620 4.27528 0 - 0.182567 2024-04-16T07:45:05.407 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 127 16 8636 8620 4.24161 0 - 0.182567 2024-04-16T07:45:05.936 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:05.937 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:05.942 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:05.942 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:45:05.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-16T07:45:06.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:06.071+0000 7fe151f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:06.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:06.082+0000 7fe151f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:06.094 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:45:06.098 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-04-16T07:45:06.098 INFO:teuthology.orchestra.run.smithi031.stderr:config get 2024-04-16T07:45:06.098 INFO:teuthology.orchestra.run.smithi031.stderr:config help 2024-04-16T07:45:06.098 INFO:teuthology.orchestra.run.smithi031.stderr:config set ... 2024-04-16T07:45:06.098 INFO:teuthology.orchestra.run.smithi031.stderr:config show 2024-04-16T07:45:06.098 INFO:teuthology.orchestra.run.smithi031.stderr:deep_scrub 2024-04-16T07:45:06.098 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_ops 2024-04-16T07:45:06.099 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_slow_ops 2024-04-16T07:45:06.099 INFO:teuthology.orchestra.run.smithi031.stderr:dump_metrics [] 2024-04-16T07:45:06.099 INFO:teuthology.orchestra.run.smithi031.stderr:dump_ops_in_flight 2024-04-16T07:45:06.099 INFO:teuthology.orchestra.run.smithi031.stderr:dump_pgstate_history 2024-04-16T07:45:06.099 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-04-16T07:45:06.099 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:06.100 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:06.102 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:45:06.102 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:45:06.102 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-16T07:45:06.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:06.252+0000 7fba94ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:06.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:06.263+0000 7fba94ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:06.279 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-04-16T07:45:06.279 INFO:teuthology.orchestra.run.smithi031.stderr:config get 2024-04-16T07:45:06.279 INFO:teuthology.orchestra.run.smithi031.stderr:config help 2024-04-16T07:45:06.279 INFO:teuthology.orchestra.run.smithi031.stderr:config set ... 2024-04-16T07:45:06.279 INFO:teuthology.orchestra.run.smithi031.stderr:config show 2024-04-16T07:45:06.279 INFO:teuthology.orchestra.run.smithi031.stderr:deep_scrub 2024-04-16T07:45:06.279 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_ops 2024-04-16T07:45:06.279 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_slow_ops 2024-04-16T07:45:06.279 INFO:teuthology.orchestra.run.smithi031.stderr:dump_metrics [] 2024-04-16T07:45:06.279 INFO:teuthology.orchestra.run.smithi031.stderr:dump_ops_in_flight 2024-04-16T07:45:06.279 INFO:teuthology.orchestra.run.smithi031.stderr:dump_pgstate_history 2024-04-16T07:45:06.279 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-04-16T07:45:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:45:06.282 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:45:06.282 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-16T07:45:06.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:06.394+0000 7fac3615d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:06.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:06.403+0000 7fac3615d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:06.407 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 128 16 8636 8620 4.20848 0 - 0.182567 2024-04-16T07:45:06.409 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:45:06.412 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:06.413 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:45:07.408 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 129 16 8646 8630 4.1807 0.0520833 0.0246901 0.213831 2024-04-16T07:45:08.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:45:08.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:08.225+0000 7faed4fd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:08.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:08.232+0000 7faed4fd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:08.238 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:08.281 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:45:08.285 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:08.285 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:08.407 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 130 16 8646 8630 4.14854 0 - 0.213831 2024-04-16T07:45:09.408 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 131 16 8646 8630 4.11687 0 - 0.213831 2024-04-16T07:45:10.286 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:45:10.408 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 132 16 8646 8630 4.08568 0 - 0.213831 2024-04-16T07:45:10.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:10.412+0000 7fbebe75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:10.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:10.419+0000 7fbebe75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:10.425 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:10.468 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:45:10.471 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:10.472 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:11.408 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 133 16 8646 8630 4.05496 0 - 0.213831 2024-04-16T07:45:11.413 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3, 0] live_osds: [2, 1] 2024-04-16T07:45:11.413 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:45:11.413 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-16T07:45:11.414 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-16T07:45:11.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:11.537+0000 7fa96266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:11.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:11.547+0000 7fa96266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:11.553 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:11.556 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:11.556 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:45:11.556 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-16T07:45:11.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:11.667+0000 7ff3b0cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:11.677+0000 7ff3b0cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:11.694 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-04-16T07:45:11.695 INFO:teuthology.orchestra.run.smithi031.stderr:config get 2024-04-16T07:45:11.695 INFO:teuthology.orchestra.run.smithi031.stderr:config help 2024-04-16T07:45:11.695 INFO:teuthology.orchestra.run.smithi031.stderr:config set ... 2024-04-16T07:45:11.695 INFO:teuthology.orchestra.run.smithi031.stderr:config show 2024-04-16T07:45:11.695 INFO:teuthology.orchestra.run.smithi031.stderr:deep_scrub 2024-04-16T07:45:11.695 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_ops 2024-04-16T07:45:11.695 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_slow_ops 2024-04-16T07:45:11.695 INFO:teuthology.orchestra.run.smithi031.stderr:dump_metrics [] 2024-04-16T07:45:11.695 INFO:teuthology.orchestra.run.smithi031.stderr:dump_ops_in_flight 2024-04-16T07:45:11.695 INFO:teuthology.orchestra.run.smithi031.stderr:dump_pgstate_history 2024-04-16T07:45:11.695 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-04-16T07:45:11.698 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:45:11.698 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:45:11.698 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-16T07:45:11.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:11.809+0000 7faf10f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:11.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:11.819+0000 7faf10f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:11.836 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-04-16T07:45:11.837 INFO:teuthology.orchestra.run.smithi031.stderr:config get 2024-04-16T07:45:11.837 INFO:teuthology.orchestra.run.smithi031.stderr:config help 2024-04-16T07:45:11.837 INFO:teuthology.orchestra.run.smithi031.stderr:config set ... 2024-04-16T07:45:11.837 INFO:teuthology.orchestra.run.smithi031.stderr:config show 2024-04-16T07:45:11.837 INFO:teuthology.orchestra.run.smithi031.stderr:deep_scrub 2024-04-16T07:45:11.837 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_ops 2024-04-16T07:45:11.837 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_slow_ops 2024-04-16T07:45:11.837 INFO:teuthology.orchestra.run.smithi031.stderr:dump_metrics [] 2024-04-16T07:45:11.837 INFO:teuthology.orchestra.run.smithi031.stderr:dump_ops_in_flight 2024-04-16T07:45:11.837 INFO:teuthology.orchestra.run.smithi031.stderr:dump_pgstate_history 2024-04-16T07:45:11.837 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-04-16T07:45:11.839 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:45:11.840 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:45:11.840 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-16T07:45:11.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:11.951+0000 7efde8f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:11.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:11.960+0000 7efde8f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:11.966 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:45:11.970 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:11.970 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:45:12.408 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 134 16 8646 8630 4.0247 0 - 0.213831 2024-04-16T07:45:12.473 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:45:12.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:12.592+0000 7fbf2ad5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:12.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:12.598+0000 7fbf2ad5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:12.605 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:12.647 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:45:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:12.651 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:13.408 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 135 16 8646 8630 3.99489 0 - 0.213831 2024-04-16T07:45:14.408 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 136 16 8646 8630 3.96551 0 - 0.213831 2024-04-16T07:45:14.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:45:14.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:14.776+0000 7fa942554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:14.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:14.782+0000 7fa942554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:14.789 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:14.841 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:45:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:14.844 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:15.408 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 137 16 8646 8630 3.93657 0 - 0.213831 2024-04-16T07:45:16.409 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 138 16 8646 8630 3.90804 0 - 0.213831 2024-04-16T07:45:16.846 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:45:16.971 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3, 0] live_osds: [2, 1] 2024-04-16T07:45:16.971 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:45:16.971 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-04-16T07:45:16.971 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-16T07:45:16.971 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-16T07:45:16.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:16.971+0000 7fef340ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:16.974 INFO:tasks.ceph.osd.3:Started 2024-04-16T07:45:16.975 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:45:16.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:16.978+0000 7fef340ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:16.983 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:17.027 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:45:17.031 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:17.031 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:17.091 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:45:17.094 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:45:17.095 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:45:17.409 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 139 16 8646 8630 3.87993 0 - 0.213831 2024-04-16T07:45:18.409 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:2024-04-16T07:45:18.410605+0000 min lat: 0.0165093 max lat: 41.3478 avg lat: 0.213831 2024-04-16T07:45:18.409 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:45:18.409 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 140 16 8646 8630 3.85221 0 - 0.213831 2024-04-16T07:45:19.032 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:45:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:19.151+0000 7fe81482d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:19.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:19.158+0000 7fe81482d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:19.163 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:19.204 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:45:19.208 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:19.208 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:19.409 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 141 16 8646 8630 3.82489 0 - 0.213831 2024-04-16T07:45:20.410 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 142 16 8646 8630 3.79795 0 - 0.213831 2024-04-16T07:45:21.209 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:45:21.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:21.337+0000 7ffaadb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:21.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:21.343+0000 7ffaadb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:21.348 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:21.390 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:45:21.394 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:21.394 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:21.409 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 143 16 8646 8630 3.7714 0 - 0.213831 2024-04-16T07:45:22.096 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:45:22.201 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:45:22.203 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:45:22.204 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:45:22.410 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 144 16 8646 8630 3.74521 0 - 0.213831 2024-04-16T07:45:23.395 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:45:23.409 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 145 16 8646 8630 3.71938 0 - 0.213831 2024-04-16T07:45:23.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:23.521+0000 7f5fe099d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:23.527+0000 7f5fe099d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:23.575 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:45:23.578 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:23.579 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:24.410 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 146 16 8646 8630 3.6939 0 - 0.213831 2024-04-16T07:45:25.410 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 147 16 8646 8630 3.66877 0 - 0.213831 2024-04-16T07:45:25.580 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:45:25.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:25.706+0000 7f0a5c55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:25.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:25.712+0000 7f0a5c55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:25.718 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:25.759 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:45:25.763 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:25.763 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:26.410 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 148 16 8646 8630 3.64398 0 - 0.213831 2024-04-16T07:45:27.205 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:45:27.311 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:45:27.314 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:45:27.314 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:45:27.410 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 149 16 8646 8630 3.61952 0 - 0.213831 2024-04-16T07:45:27.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:45:27.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:27.889+0000 7f8bc3ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:27.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:27.896+0000 7f8bc3ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:27.901 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:27.942 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:45:27.945 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:27.946 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:28.410 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 150 16 8646 8630 3.59539 0 - 0.213831 2024-04-16T07:45:29.410 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 151 16 8646 8630 3.57158 0 - 0.213831 2024-04-16T07:45:29.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:45:30.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:30.071+0000 7f7175e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:30.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:30.077+0000 7f7175e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:30.082 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:30.124 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:45:30.127 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:30.128 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:30.410 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 152 16 8646 8630 3.54809 0 - 0.213831 2024-04-16T07:45:31.411 INFO:tasks.radosbench.radosbench.0.smithi031.stdout: 153 16 8646 8630 3.5249 0 - 0.213831 2024-04-16T07:45:32.129 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:45:32.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:32.273+0000 7f998802d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:32.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:32.280+0000 7f998802d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:32.285 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:32.315 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:45:32.411 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Total time run: 153.987 2024-04-16T07:45:32.411 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Total writes made: 8646 2024-04-16T07:45:32.411 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Write size: 65536 2024-04-16T07:45:32.411 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Object size: 65536 2024-04-16T07:45:32.411 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Bandwidth (MB/sec): 3.50923 2024-04-16T07:45:32.411 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Stddev Bandwidth: 5.63321 2024-04-16T07:45:32.411 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Max bandwidth (MB/sec): 13.9375 2024-04-16T07:45:32.411 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Min bandwidth (MB/sec): 0 2024-04-16T07:45:32.411 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Average IOPS: 56 2024-04-16T07:45:32.411 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Stddev IOPS: 90.1406 2024-04-16T07:45:32.412 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Max IOPS: 223 2024-04-16T07:45:32.412 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Min IOPS: 0 2024-04-16T07:45:32.412 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Average Latency(s): 0.284803 2024-04-16T07:45:32.412 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Stddev Latency(s): 2.62608 2024-04-16T07:45:32.412 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Max latency(s): 66.495 2024-04-16T07:45:32.412 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Min latency(s): 0.0165093 2024-04-16T07:45:32.467 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Cleaning up (deleting benchmark objects) 2024-04-16T07:45:32.505 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-16T07:45:32.505 INFO:teuthology.orchestra.run.smithi094.stdout: "ops_in_flight": [ 2024-04-16T07:45:32.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:45:32.505 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:45:32.506 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 214, 2024-04-16T07:45:32.506 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:45:32.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:45:32.506 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:32.506 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.472793+0000" 2024-04-16T07:45:32.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.506 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.506 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:32.506 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.472807+0000" 2024-04-16T07:45:32.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.506 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.506 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:32.507 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.475045+0000" 2024-04-16T07:45:32.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.507 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.507 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:32.507 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.507 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.507 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:32.507 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.477339+0000" 2024-04-16T07:45:32.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.507 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.507 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:32.507 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.478116+0000" 2024-04-16T07:45:32.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.507 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.508 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:32.508 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:45:32.508 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:32.508 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.508 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.508 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:45:32.508 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:45:32.508 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 217, 2024-04-16T07:45:32.508 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:45:32.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:45:32.508 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:32.509 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.485187+0000" 2024-04-16T07:45:32.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.509 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.509 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:32.509 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.485198+0000" 2024-04-16T07:45:32.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.509 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.509 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:32.509 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.487974+0000" 2024-04-16T07:45:32.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.509 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.509 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:32.509 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.510 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.510 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:32.510 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.488525+0000" 2024-04-16T07:45:32.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.510 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.510 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:32.510 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.489279+0000" 2024-04-16T07:45:32.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.510 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.510 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:32.510 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:45:32.510 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:32.511 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.511 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.511 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:45:32.511 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "peering_event", 2024-04-16T07:45:32.511 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 216, 2024-04-16T07:45:32.511 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:45:32.511 INFO:teuthology.orchestra.run.smithi094.stdout: "PeeringEvent": { 2024-04-16T07:45:32.511 INFO:teuthology.orchestra.run.smithi094.stdout: "from": "3", 2024-04-16T07:45:32.511 INFO:teuthology.orchestra.run.smithi094.stdout: "pgid": "3.0", 2024-04-16T07:45:32.511 INFO:teuthology.orchestra.run.smithi094.stdout: "sent": 29, 2024-04-16T07:45:32.511 INFO:teuthology.orchestra.run.smithi094.stdout: "requested": 29, 2024-04-16T07:45:32.511 INFO:teuthology.orchestra.run.smithi094.stdout: "evt": "epoch_sent: 29 epoch_requested: 29 AllReplicasRecovered", 2024-04-16T07:45:32.511 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-04-16T07:45:32.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:45:32.512 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:32.512 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:45:32.512 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T07:45:32.512 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.484956+0000" 2024-04-16T07:45:32.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:45:32.512 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T07:45:32.512 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:45:32.512 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T07:45:32.512 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.485444+0000" 2024-04-16T07:45:32.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:45:32.513 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:32.513 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.513 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.513 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-04-16T07:45:32.513 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.513 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:45:32.513 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:45:32.513 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 15, 2024-04-16T07:45:32.513 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:45:32.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:45:32.513 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:32.514 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.955008+0000" 2024-04-16T07:45:32.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.514 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.514 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:32.514 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.955032+0000" 2024-04-16T07:45:32.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.514 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.514 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:32.514 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.972214+0000" 2024-04-16T07:45:32.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.514 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.514 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:32.514 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.972581+0000" 2024-04-16T07:45:32.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.514 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.515 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:32.515 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:31.149689+0000" 2024-04-16T07:45:32.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.515 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.515 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:32.515 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:31.149894+0000" 2024-04-16T07:45:32.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.515 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.515 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:32.515 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:45:32.515 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:32.515 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.125124+0000" 2024-04-16T07:45:32.515 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.516 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:45:32.516 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:45:32.516 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 17, 2024-04-16T07:45:32.516 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:45:32.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:45:32.516 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:32.516 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.956021+0000" 2024-04-16T07:45:32.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.516 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.516 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:32.516 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.956028+0000" 2024-04-16T07:45:32.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.516 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.516 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:32.517 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.972735+0000" 2024-04-16T07:45:32.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.517 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.517 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:32.517 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.517 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.517 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:32.517 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:31.150850+0000" 2024-04-16T07:45:32.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.517 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.517 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:32.517 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:31.151522+0000" 2024-04-16T07:45:32.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.517 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.518 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:32.518 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:45:32.518 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:32.518 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.296921+0000" 2024-04-16T07:45:32.518 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.518 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:45:32.518 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:45:32.518 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 18, 2024-04-16T07:45:32.518 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:45:32.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:45:32.518 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:32.519 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.956293+0000" 2024-04-16T07:45:32.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.519 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.519 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:32.519 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.956299+0000" 2024-04-16T07:45:32.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.519 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.519 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:32.519 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.972825+0000" 2024-04-16T07:45:32.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.519 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.521 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:32.521 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.521 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.521 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:32.521 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:31.151292+0000" 2024-04-16T07:45:32.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.521 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.521 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:32.521 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:31.152171+0000" 2024-04-16T07:45:32.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.522 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.522 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:32.522 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:45:32.522 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:32.522 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.307613+0000" 2024-04-16T07:45:32.522 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.522 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:45:32.522 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:45:32.522 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 19, 2024-04-16T07:45:32.522 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:45:32.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:45:32.523 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:32.523 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.956698+0000" 2024-04-16T07:45:32.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.523 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.523 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:32.523 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.956705+0000" 2024-04-16T07:45:32.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.523 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.523 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:32.523 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.972900+0000" 2024-04-16T07:45:32.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.523 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.523 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:32.524 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.524 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.524 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:32.524 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:31.151734+0000" 2024-04-16T07:45:32.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.524 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.524 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:32.524 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:31.152984+0000" 2024-04-16T07:45:32.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.524 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.524 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:32.524 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:45:32.524 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:32.524 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.312059+0000" 2024-04-16T07:45:32.525 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.525 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:45:32.525 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:45:32.525 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 21, 2024-04-16T07:45:32.525 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:45:32.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:45:32.525 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:32.525 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.957336+0000" 2024-04-16T07:45:32.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.525 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.525 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:32.526 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.957343+0000" 2024-04-16T07:45:32.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.526 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.526 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:32.526 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.973046+0000" 2024-04-16T07:45:32.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.526 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.526 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:32.526 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.526 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.526 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:32.526 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:31.154110+0000" 2024-04-16T07:45:32.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.527 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.527 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:32.527 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:31.155290+0000" 2024-04-16T07:45:32.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.527 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.527 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:32.527 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:45:32.527 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:32.527 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.314397+0000" 2024-04-16T07:45:32.527 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.529 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:45:32.529 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "background_recovery_sub", 2024-04-16T07:45:32.530 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 215, 2024-04-16T07:45:32.530 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": {} 2024-04-16T07:45:32.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:45:32.530 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:45:32.530 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 16, 2024-04-16T07:45:32.530 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:45:32.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:45:32.530 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:32.530 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.955525+0000" 2024-04-16T07:45:32.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.530 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.530 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:32.530 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.955539+0000" 2024-04-16T07:45:32.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.531 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.531 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:32.531 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.972337+0000" 2024-04-16T07:45:32.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.531 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.531 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:32.531 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.531 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.531 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:32.531 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:31.150029+0000" 2024-04-16T07:45:32.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.531 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.531 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:32.531 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:31.151433+0000" 2024-04-16T07:45:32.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.532 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.532 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:32.532 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:45:32.532 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:32.532 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.328624+0000" 2024-04-16T07:45:32.532 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.532 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:45:32.532 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:45:32.532 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 20, 2024-04-16T07:45:32.532 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:45:32.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:45:32.533 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:32.533 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.956961+0000" 2024-04-16T07:45:32.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.533 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.533 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:32.533 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.956968+0000" 2024-04-16T07:45:32.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.533 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.533 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:32.533 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:30.972976+0000" 2024-04-16T07:45:32.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.533 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.533 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:32.534 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.534 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.534 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:32.534 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:31.152835+0000" 2024-04-16T07:45:32.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.534 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.534 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:32.534 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:31.156313+0000" 2024-04-16T07:45:32.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.534 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:45:32.534 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:32.534 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:32.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:45:32.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:45:32.535 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:32.535 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:45:32.330119+0000" 2024-04-16T07:45:32.535 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.535 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.535 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:45:32.535 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-04-16T07:45:32.535 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-16T07:45:32.542 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:32.543 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:34.544 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:45:34.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:34.668+0000 7fee6e66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:34.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:34.676+0000 7fee6e66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:34.682 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:34.940 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:34.940 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:36.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:45:37.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:37.074+0000 7f3b4bb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:37.082+0000 7f3b4bb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.089 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:37.451 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:37.452 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:37.537 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0] live_osds: [2, 1, 3] 2024-04-16T07:45:37.537 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:45:37.537 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-04-16T07:45:37.538 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-16T07:45:37.538 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-16T07:45:37.540 INFO:tasks.ceph.osd.0:Started 2024-04-16T07:45:37.540 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-16T07:45:37.651 INFO:teuthology.orchestra.run.smithi031.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:45:37.654 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:45:37.655 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-16T07:45:39.452 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:45:39.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:39.581+0000 7f645b664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:39.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:39.589+0000 7f645b664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:39.595 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:40.001 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:40.001 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:42.002 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:45:42.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:42.163+0000 7fdf9062d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:42.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:42.172+0000 7fdf9062d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:42.180 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:42.472 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:42.472 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:42.656 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-16T07:45:42.771 INFO:teuthology.orchestra.run.smithi031.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:45:42.774 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:45:42.774 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-16T07:45:44.473 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:45:44.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:44.596+0000 7f656a22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:44.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:44.604+0000 7f656a22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:44.610 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:44.842 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:44.842 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:46.843 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:45:47.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:47.003+0000 7f9606c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:47.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:47.013+0000 7f9606c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:47.022 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:45:47.378 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:45:47.378 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:45:47.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-16T07:45:47.956 INFO:teuthology.orchestra.run.smithi031.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:45:47.960 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:45:47.960 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-16T07:45:49.379 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:45:49.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:49.493+0000 7fb788bfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:49.499+0000 7fb788bfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:51.597 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:45:51.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:51.764+0000 7f193306d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:51.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:51.774+0000 7f193306d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:52.961 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-16T07:45:53.166 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-16T07:45:53.166 INFO:teuthology.orchestra.run.smithi031.stdout: "ops_in_flight": [ 2024-04-16T07:45:53.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:45:53.166 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "replicated_request", 2024-04-16T07:45:53.166 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 256, 2024-04-16T07:45:53.166 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:45:53.166 INFO:teuthology.orchestra.run.smithi031.stdout: "RepRequest": { 2024-04-16T07:45:53.166 INFO:teuthology.orchestra.run.smithi031.stdout: "reqid": "client.4276.0:11745", 2024-04-16T07:45:53.166 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "3.f", 2024-04-16T07:45:53.166 INFO:teuthology.orchestra.run.smithi031.stdout: "map_epoch": 32, 2024-04-16T07:45:53.166 INFO:teuthology.orchestra.run.smithi031.stdout: "min_epoch": 31, 2024-04-16T07:45:53.166 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "3:f2af6ceb:::benchmark_data_smithi031_32265_object3096:head", 2024-04-16T07:45:53.166 INFO:teuthology.orchestra.run.smithi031.stdout: "from": "1" 2024-04-16T07:45:53.167 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.167 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:45:53.167 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "replicated_request", 2024-04-16T07:45:53.167 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 33554589, 2024-04-16T07:45:53.167 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:45:53.167 INFO:teuthology.orchestra.run.smithi031.stdout: "RepRequest": { 2024-04-16T07:45:53.167 INFO:teuthology.orchestra.run.smithi031.stdout: "reqid": "client.4276.0:11746", 2024-04-16T07:45:53.167 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "3.3", 2024-04-16T07:45:53.167 INFO:teuthology.orchestra.run.smithi031.stdout: "map_epoch": 32, 2024-04-16T07:45:53.167 INFO:teuthology.orchestra.run.smithi031.stdout: "min_epoch": 31, 2024-04-16T07:45:53.167 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "3:c29882cb:::benchmark_data_smithi031_32265_object3097:head", 2024-04-16T07:45:53.167 INFO:teuthology.orchestra.run.smithi031.stdout: "from": "2" 2024-04-16T07:45:53.167 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.167 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:45:53.168 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "replicated_request", 2024-04-16T07:45:53.168 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 16777454, 2024-04-16T07:45:53.168 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:45:53.168 INFO:teuthology.orchestra.run.smithi031.stdout: "RepRequest": { 2024-04-16T07:45:53.168 INFO:teuthology.orchestra.run.smithi031.stdout: "reqid": "client.4276.0:11739", 2024-04-16T07:45:53.168 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "3.1", 2024-04-16T07:45:53.168 INFO:teuthology.orchestra.run.smithi031.stdout: "map_epoch": 32, 2024-04-16T07:45:53.168 INFO:teuthology.orchestra.run.smithi031.stdout: "min_epoch": 31, 2024-04-16T07:45:53.168 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "3:8ab219af:::benchmark_data_smithi031_32265_object3090:head", 2024-04-16T07:45:53.168 INFO:teuthology.orchestra.run.smithi031.stdout: "from": "3" 2024-04-16T07:45:53.168 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.168 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:45:53.169 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "background_recovery_sub", 2024-04-16T07:45:53.169 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 16777452, 2024-04-16T07:45:53.169 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": {} 2024-04-16T07:45:53.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:45:53.169 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:45:53.169 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 57, 2024-04-16T07:45:53.169 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:45:53.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.169 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:53.169 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.236844+0000" 2024-04-16T07:45:53.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.169 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.169 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:53.169 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.236854+0000" 2024-04-16T07:45:53.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.170 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.170 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:53.170 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.237147+0000" 2024-04-16T07:45:53.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.170 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.170 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:53.170 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.170 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.170 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:53.170 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.237721+0000" 2024-04-16T07:45:53.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.170 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.170 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:53.171 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.237918+0000" 2024-04-16T07:45:53.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.171 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.171 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:53.171 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.171 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:53.171 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.957620+0000" 2024-04-16T07:45:53.171 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.171 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:45:53.171 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:45:53.171 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 47, 2024-04-16T07:45:53.171 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:45:53.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.172 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:53.172 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.212625+0000" 2024-04-16T07:45:53.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.172 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.172 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:53.172 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.212633+0000" 2024-04-16T07:45:53.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.172 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.172 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:53.172 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.212853+0000" 2024-04-16T07:45:53.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.172 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.172 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:53.172 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.173 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.173 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:53.173 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.212954+0000" 2024-04-16T07:45:53.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.173 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.173 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:53.173 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.213063+0000" 2024-04-16T07:45:53.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.173 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.173 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:53.173 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.173 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:53.173 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.966520+0000" 2024-04-16T07:45:53.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:45:53.174 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:45:53.174 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 52, 2024-04-16T07:45:53.174 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:45:53.174 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.174 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:53.174 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.221633+0000" 2024-04-16T07:45:53.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.174 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.174 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:53.174 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.221642+0000" 2024-04-16T07:45:53.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.174 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.175 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:53.175 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.221813+0000" 2024-04-16T07:45:53.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.175 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.175 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:53.175 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.175 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.175 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:53.175 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.222600+0000" 2024-04-16T07:45:53.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.175 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.175 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:53.175 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.223677+0000" 2024-04-16T07:45:53.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.175 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.176 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:53.176 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.176 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:53.176 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.966616+0000" 2024-04-16T07:45:53.176 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.176 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:45:53.176 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:45:53.176 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 59, 2024-04-16T07:45:53.176 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:45:53.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.176 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:53.176 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.242476+0000" 2024-04-16T07:45:53.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.177 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.177 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:53.177 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.242492+0000" 2024-04-16T07:45:53.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.177 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.177 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:53.177 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.243920+0000" 2024-04-16T07:45:53.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.177 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.177 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:53.177 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.177 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.177 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:53.177 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.244246+0000" 2024-04-16T07:45:53.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.178 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.178 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:53.178 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.245694+0000" 2024-04-16T07:45:53.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.178 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.178 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:53.178 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.178 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:53.178 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.966724+0000" 2024-04-16T07:45:53.178 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.178 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:45:53.179 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "replicated_request", 2024-04-16T07:45:53.179 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 16777455, 2024-04-16T07:45:53.179 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:45:53.179 INFO:teuthology.orchestra.run.smithi031.stdout: "RepRequest": { 2024-04-16T07:45:53.179 INFO:teuthology.orchestra.run.smithi031.stdout: "reqid": "client.4276.0:11742", 2024-04-16T07:45:53.179 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "3.7", 2024-04-16T07:45:53.179 INFO:teuthology.orchestra.run.smithi031.stdout: "map_epoch": 32, 2024-04-16T07:45:53.179 INFO:teuthology.orchestra.run.smithi031.stdout: "min_epoch": 31, 2024-04-16T07:45:53.179 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "3:ebbb1df4:::benchmark_data_smithi031_32265_object3093:head", 2024-04-16T07:45:53.179 INFO:teuthology.orchestra.run.smithi031.stdout: "from": "3" 2024-04-16T07:45:53.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:45:53.179 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "replicated_request", 2024-04-16T07:45:53.180 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 16777456, 2024-04-16T07:45:53.180 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:45:53.180 INFO:teuthology.orchestra.run.smithi031.stdout: "RepRequest": { 2024-04-16T07:45:53.180 INFO:teuthology.orchestra.run.smithi031.stdout: "reqid": "client.4276.0:11744", 2024-04-16T07:45:53.180 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "3.7", 2024-04-16T07:45:53.180 INFO:teuthology.orchestra.run.smithi031.stdout: "map_epoch": 32, 2024-04-16T07:45:53.180 INFO:teuthology.orchestra.run.smithi031.stdout: "min_epoch": 31, 2024-04-16T07:45:53.180 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "3:e757077b:::benchmark_data_smithi031_32265_object3095:head", 2024-04-16T07:45:53.180 INFO:teuthology.orchestra.run.smithi031.stdout: "from": "3" 2024-04-16T07:45:53.180 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.180 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:45:53.180 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "replicated_request", 2024-04-16T07:45:53.180 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 33554590, 2024-04-16T07:45:53.181 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:45:53.181 INFO:teuthology.orchestra.run.smithi031.stdout: "RepRequest": { 2024-04-16T07:45:53.181 INFO:teuthology.orchestra.run.smithi031.stdout: "reqid": "client.4276.0:11747", 2024-04-16T07:45:53.181 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "3.c", 2024-04-16T07:45:53.181 INFO:teuthology.orchestra.run.smithi031.stdout: "map_epoch": 32, 2024-04-16T07:45:53.181 INFO:teuthology.orchestra.run.smithi031.stdout: "min_epoch": 31, 2024-04-16T07:45:53.181 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "3:3244c47e:::benchmark_data_smithi031_32265_object3098:head", 2024-04-16T07:45:53.181 INFO:teuthology.orchestra.run.smithi031.stdout: "from": "2" 2024-04-16T07:45:53.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:45:53.181 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "background_recovery_sub", 2024-04-16T07:45:53.181 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 16777457, 2024-04-16T07:45:53.181 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": {} 2024-04-16T07:45:53.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:45:53.182 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:45:53.182 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 48, 2024-04-16T07:45:53.182 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:45:53.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.182 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:53.182 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.214950+0000" 2024-04-16T07:45:53.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.182 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.182 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:53.182 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.214959+0000" 2024-04-16T07:45:53.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.182 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.182 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:53.182 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.215115+0000" 2024-04-16T07:45:53.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.183 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.183 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:53.183 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.183 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.183 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:53.183 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.215932+0000" 2024-04-16T07:45:53.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.183 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.183 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:53.183 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.216980+0000" 2024-04-16T07:45:53.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.183 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.183 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:50.014199+0000" 2024-04-16T07:45:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:45:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:45:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 46, 2024-04-16T07:45:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:45:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.210396+0000" 2024-04-16T07:45:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.210421+0000" 2024-04-16T07:45:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.210895+0000" 2024-04-16T07:45:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.211347+0000" 2024-04-16T07:45:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.212456+0000" 2024-04-16T07:45:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:50.000255+0000" 2024-04-16T07:45:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:45:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:45:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 50, 2024-04-16T07:45:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:45:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.218787+0000" 2024-04-16T07:45:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.218797+0000" 2024-04-16T07:45:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.219572+0000" 2024-04-16T07:45:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.219726+0000" 2024-04-16T07:45:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.220671+0000" 2024-04-16T07:45:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:50.000432+0000" 2024-04-16T07:45:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:45:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:45:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 54, 2024-04-16T07:45:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:45:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.225108+0000" 2024-04-16T07:45:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:53.190 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.225119+0000" 2024-04-16T07:45:53.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.190 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.190 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:53.190 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.225622+0000" 2024-04-16T07:45:53.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.190 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.190 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:53.190 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.190 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.190 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:53.190 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.226735+0000" 2024-04-16T07:45:53.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.191 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.191 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:53.191 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.232536+0000" 2024-04-16T07:45:53.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.191 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.191 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:53.191 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.191 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.191 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:53.191 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:50.000595+0000" 2024-04-16T07:45:53.191 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.191 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:45:53.191 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:45:53.192 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 55, 2024-04-16T07:45:53.192 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:45:53.192 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.192 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:53.192 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.231592+0000" 2024-04-16T07:45:53.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.192 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.192 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:53.192 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.231616+0000" 2024-04-16T07:45:53.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.192 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.192 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:53.192 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.232233+0000" 2024-04-16T07:45:53.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.192 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.192 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:53.193 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.193 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.193 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:53.193 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.233326+0000" 2024-04-16T07:45:53.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.193 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.193 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:53.193 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:49.235806+0000" 2024-04-16T07:45:53.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.193 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.193 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:53.193 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.193 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.193 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:53.193 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:50.000753+0000" 2024-04-16T07:45:53.194 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.194 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-16T07:45:53.194 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client_request", 2024-04-16T07:45:53.194 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 136, 2024-04-16T07:45:53.194 INFO:teuthology.orchestra.run.smithi031.stdout: "detail": { 2024-04-16T07:45:53.194 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.194 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:45:53.194 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:51.465452+0000" 2024-04-16T07:45:53.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.194 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.194 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:45:53.194 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:51.465471+0000" 2024-04-16T07:45:53.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.194 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.195 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:45:53.195 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:51.465640+0000" 2024-04-16T07:45:53.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.195 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.195 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:45:53.195 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.195 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.195 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:45:53.195 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:51.465768+0000" 2024-04-16T07:45:53.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.195 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.195 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:45:53.195 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:51.466035+0000" 2024-04-16T07:45:53.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.195 INFO:teuthology.orchestra.run.smithi031.stdout: "blocking_event": { 2024-04-16T07:45:53.196 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:45:53.196 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "0.000000" 2024-04-16T07:45:53.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-16T07:45:53.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time_event": { 2024-04-16T07:45:53.196 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:45:53.196 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-04-16T07:45:51.467120+0000" 2024-04-16T07:45:53.196 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.196 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.196 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-16T07:45:53.196 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-16T07:45:53.196 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-16T07:45:54.202 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:45:54.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:54.360+0000 7ff5eaaed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:54.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:54.370+0000 7ff5eaaed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:56.879 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:45:57.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:57.050+0000 7fcb7ae2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:57.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:57.058+0000 7fcb7ae2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:58.198 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 1, 3, 0] 2024-04-16T07:45:58.198 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:45:58.198 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:45:58.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:58.376+0000 7f50454ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:58.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:58.393+0000 7f50454ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:58.741 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-16T07:45:58.741 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-16T07:45:58.758 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-04-16T07:45:58.012630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220135992,"num_objects":3353,"num_object_clones":0,"num_object_copies":10059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":10027,"num_write_kb":417383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":484,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12312,"ondisk_log_size":12312,"up":75,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":206752,"kb_used_data":71100,"kb_used_omap":0,"kb_used_meta":135616,"kb_avail":377280608,"statfs":{"total":386547056640,"available":386335342592,"internally_reserved":0,"allocated":72806400,"data_stored":1146933104,"data_compressed":54449721,"data_compressed_allocated":71671808,"data_compressed_original":1146503168,"omap_allocated":0,"internal_metadata":138870784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-58589184,"num_objects":-894,"num_object_clones":0,"num_object_copies":-2682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-653,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1000,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":458752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.062567"},"pg_stats":[{"pgid":"3.f","version":"32'742","reported_seq":727,"reported_epoch":32,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-16T07:45:56.864973+0000","last_change":"2024-04-16T07:45:51.840898+0000","last_active":"2024-04-16T07:45:56.864973+0000","last_peered":"2024-04-16T07:45:56.864973+0000","last_clean":"2024-04-16T07:43:48.494596+0000","last_became_active":"2024-04-16T07:45:51.366885+0000","last_became_peered":"2024-04-16T07:45:51.366885+0000","last_unstale":"2024-04-16T07:45:56.864973+0000","last_undegraded":"2024-04-16T07:45:56.864973+0000","last_fullsized":"2024-04-16T07:45:50.991119+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":742,"log_dups_size":0,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832704,"num_objects":89,"num_object_clones":0,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":531,"num_write_kb":19840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":89}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"32'706","reported_seq":786,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-16T07:45:54.899689+0000","last_change":"2024-04-16T07:45:32.654189+0000","last_active":"2024-04-16T07:45:54.899689+0000","last_peered":"2024-04-16T07:45:54.899689+0000","last_clean":"2024-04-16T07:45:54.899689+0000","last_became_active":"2024-04-16T07:45:32.115728+0000","last_became_peered":"2024-04-16T07:45:32.115728+0000","last_unstale":"2024-04-16T07:45:54.899689+0000","last_undegraded":"2024-04-16T07:45:54.899689+0000","last_fullsized":"2024-04-16T07:45:54.899689+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19726336,"num_objects":301,"num_object_clones":0,"num_object_copies":903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":707,"num_write_kb":32256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"32'744","reported_seq":809,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-16T07:45:56.789606+0000","last_change":"2024-04-16T07:45:51.431117+0000","last_active":"2024-04-16T07:45:56.789606+0000","last_peered":"2024-04-16T07:45:56.789606+0000","last_clean":"2024-04-16T07:44:25.827250+0000","last_became_active":"2024-04-16T07:45:51.427342+0000","last_became_peered":"2024-04-16T07:45:51.427342+0000","last_unstale":"2024-04-16T07:45:56.789606+0000","last_undegraded":"2024-04-16T07:45:51.093828+0000","last_fullsized":"2024-04-16T07:45:51.053627+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":744,"log_dups_size":0,"ondisk_log_size":744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19267608,"num_objects":295,"num_object_clones":0,"num_object_copies":885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":745,"num_write_kb":33217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":292}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"32'775","reported_seq":56,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-16T07:45:55.175220+0000","last_change":"2024-04-16T07:45:51.383957+0000","last_active":"2024-04-16T07:45:55.175220+0000","last_peered":"2024-04-16T07:45:55.175220+0000","last_clean":"2024-04-16T07:42:56.287435+0000","last_became_active":"2024-04-16T07:45:51.380228+0000","last_became_peered":"2024-04-16T07:45:51.380228+0000","last_unstale":"2024-04-16T07:45:55.175220+0000","last_undegraded":"2024-04-16T07:45:51.017322+0000","last_fullsized":"2024-04-16T07:45:50.998590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":775,"log_dups_size":0,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"32'769","reported_seq":821,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-16T07:45:56.609572+0000","last_change":"2024-04-16T07:45:51.409095+0000","last_active":"2024-04-16T07:45:56.609572+0000","last_peered":"2024-04-16T07:45:56.609572+0000","last_clean":"2024-04-16T07:44:25.649155+0000","last_became_active":"2024-04-16T07:45:51.401923+0000","last_became_peered":"2024-04-16T07:45:51.401923+0000","last_unstale":"2024-04-16T07:45:56.609572+0000","last_undegraded":"2024-04-16T07:45:51.066933+0000","last_fullsized":"2024-04-16T07:45:51.046069+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":769,"log_dups_size":0,"ondisk_log_size":769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5767168,"num_objects":88,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":528,"num_write_kb":19712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":87}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"32'790","reported_seq":628,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-16T07:45:56.693161+0000","last_change":"2024-04-16T07:45:51.438495+0000","last_active":"2024-04-16T07:45:56.693161+0000","last_peered":"2024-04-16T07:45:56.693161+0000","last_clean":"2024-04-16T07:43:48.510306+0000","last_became_active":"2024-04-16T07:45:51.429888+0000","last_became_peered":"2024-04-16T07:45:51.429888+0000","last_unstale":"2024-04-16T07:45:56.693161+0000","last_undegraded":"2024-04-16T07:45:51.071411+0000","last_fullsized":"2024-04-16T07:45:51.048168+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":790,"log_dups_size":0,"ondisk_log_size":790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6488064,"num_objects":99,"num_object_clones":0,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":581,"num_write_kb":21760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":98}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"32'740","reported_seq":243,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-16T07:45:56.704152+0000","last_change":"2024-04-16T07:45:51.451189+0000","last_active":"2024-04-16T07:45:56.704152+0000","last_peered":"2024-04-16T07:45:56.704152+0000","last_clean":"2024-04-16T07:42:56.287435+0000","last_became_active":"2024-04-16T07:45:51.442577+0000","last_became_peered":"2024-04-16T07:45:51.442577+0000","last_unstale":"2024-04-16T07:45:56.704152+0000","last_undegraded":"2024-04-16T07:45:51.044932+0000","last_fullsized":"2024-04-16T07:45:51.023138+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":740,"log_dups_size":0,"ondisk_log_size":740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"32'753","reported_seq":830,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-16T07:45:56.767794+0000","last_change":"2024-04-16T07:45:32.735314+0000","last_active":"2024-04-16T07:45:56.767794+0000","last_peered":"2024-04-16T07:45:56.767794+0000","last_clean":"2024-04-16T07:45:56.767794+0000","last_became_active":"2024-04-16T07:45:32.117948+0000","last_became_peered":"2024-04-16T07:45:32.117948+0000","last_unstale":"2024-04-16T07:45:56.767794+0000","last_undegraded":"2024-04-16T07:45:56.767794+0000","last_fullsized":"2024-04-16T07:45:56.767794+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20185088,"num_objects":308,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":754,"num_write_kb":33984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":167,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-16T07:45:51.042422+0000","last_change":"2024-04-16T07:45:50.161031+0000","last_active":"2024-04-16T07:45:51.042422+0000","last_peered":"2024-04-16T07:45:51.042422+0000","last_clean":"2024-04-16T07:45:51.042422+0000","last_became_active":"2024-04-16T07:45:50.155419+0000","last_became_peered":"2024-04-16T07:45:50.155419+0000","last_unstale":"2024-04-16T07:45:51.042422+0000","last_undegraded":"2024-04-16T07:45:51.042422+0000","last_fullsized":"2024-04-16T07:45:51.042422+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:42.229959+0000","last_clean_scrub_stamp":"2024-04-16T07:42:42.229959+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":49,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-16T07:45:51.100920+0000","last_change":"2024-04-16T07:45:50.157859+0000","last_active":"2024-04-16T07:45:51.100920+0000","last_peered":"2024-04-16T07:45:51.100920+0000","last_clean":"2024-04-16T07:45:51.100920+0000","last_became_active":"2024-04-16T07:45:50.152022+0000","last_became_peered":"2024-04-16T07:45:50.152022+0000","last_unstale":"2024-04-16T07:45:51.100920+0000","last_undegraded":"2024-04-16T07:45:51.100920+0000","last_fullsized":"2024-04-16T07:45:51.100920+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"32'790","reported_seq":1003,"reported_epoch":32,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-16T07:45:56.826423+0000","last_change":"2024-04-16T07:45:52.422112+0000","last_active":"2024-04-16T07:45:56.826423+0000","last_peered":"2024-04-16T07:45:56.826423+0000","last_clean":"2024-04-16T07:44:25.696825+0000","last_became_active":"2024-04-16T07:45:51.393672+0000","last_became_peered":"2024-04-16T07:45:51.393672+0000","last_unstale":"2024-04-16T07:45:56.826423+0000","last_undegraded":"2024-04-16T07:45:56.826423+0000","last_fullsized":"2024-04-16T07:45:50.963090+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":790,"log_dups_size":0,"ondisk_log_size":790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21299200,"num_objects":325,"num_object_clones":0,"num_object_copies":975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":791,"num_write_kb":35712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":325}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-16T07:45:51.029218+0000","last_change":"2024-04-16T07:45:50.240040+0000","last_active":"2024-04-16T07:45:51.029218+0000","last_peered":"2024-04-16T07:45:51.029218+0000","last_clean":"2024-04-16T07:45:51.029218+0000","last_became_active":"2024-04-16T07:45:50.214465+0000","last_became_peered":"2024-04-16T07:45:50.214465+0000","last_unstale":"2024-04-16T07:45:51.029218+0000","last_undegraded":"2024-04-16T07:45:51.029218+0000","last_fullsized":"2024-04-16T07:45:51.029218+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"32'851","reported_seq":1163,"reported_epoch":32,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-16T07:45:56.806860+0000","last_change":"2024-04-16T07:45:51.622169+0000","last_active":"2024-04-16T07:45:56.806860+0000","last_peered":"2024-04-16T07:45:56.806860+0000","last_clean":"2024-04-16T07:43:48.544414+0000","last_became_active":"2024-04-16T07:45:51.369818+0000","last_became_peered":"2024-04-16T07:45:51.369818+0000","last_unstale":"2024-04-16T07:45:56.806860+0000","last_undegraded":"2024-04-16T07:45:56.806860+0000","last_fullsized":"2024-04-16T07:45:51.068880+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":851,"log_dups_size":0,"ondisk_log_size":851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8323072,"num_objects":127,"num_object_clones":0,"num_object_copies":381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":635,"num_write_kb":24384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":127}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":71,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-16T07:45:51.098438+0000","last_change":"2024-04-16T07:45:32.145214+0000","last_active":"2024-04-16T07:45:51.098438+0000","last_peered":"2024-04-16T07:45:51.098438+0000","last_clean":"2024-04-16T07:45:51.098438+0000","last_became_active":"2024-04-16T07:45:32.135244+0000","last_became_peered":"2024-04-16T07:45:32.135244+0000","last_unstale":"2024-04-16T07:45:51.098438+0000","last_undegraded":"2024-04-16T07:45:51.098438+0000","last_fullsized":"2024-04-16T07:45:51.098438+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"32'779","reported_seq":840,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-16T07:45:56.638020+0000","last_change":"2024-04-16T07:45:32.487307+0000","last_active":"2024-04-16T07:45:56.638020+0000","last_peered":"2024-04-16T07:45:56.638020+0000","last_clean":"2024-04-16T07:45:56.638020+0000","last_became_active":"2024-04-16T07:45:32.128956+0000","last_became_peered":"2024-04-16T07:45:32.128956+0000","last_unstale":"2024-04-16T07:45:56.638020+0000","last_undegraded":"2024-04-16T07:45:56.638020+0000","last_fullsized":"2024-04-16T07:45:56.638020+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":779,"log_dups_size":0,"ondisk_log_size":779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7012352,"num_objects":107,"num_object_clones":0,"num_object_copies":321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":583,"num_write_kb":22080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-16T07:45:51.030065+0000","last_change":"2024-04-16T07:45:50.242860+0000","last_active":"2024-04-16T07:45:51.030065+0000","last_peered":"2024-04-16T07:45:51.030065+0000","last_clean":"2024-04-16T07:45:51.030065+0000","last_became_active":"2024-04-16T07:45:50.235023+0000","last_became_peered":"2024-04-16T07:45:50.235023+0000","last_unstale":"2024-04-16T07:45:51.030065+0000","last_undegraded":"2024-04-16T07:45:51.030065+0000","last_fullsized":"2024-04-16T07:45:51.030065+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"32'708","reported_seq":770,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-16T07:45:55.026446+0000","last_change":"2024-04-16T07:45:51.386126+0000","last_active":"2024-04-16T07:45:55.026446+0000","last_peered":"2024-04-16T07:45:55.026446+0000","last_clean":"2024-04-16T07:44:25.775407+0000","last_became_active":"2024-04-16T07:45:51.380709+0000","last_became_peered":"2024-04-16T07:45:51.380709+0000","last_unstale":"2024-04-16T07:45:55.026446+0000","last_undegraded":"2024-04-16T07:45:50.985597+0000","last_fullsized":"2024-04-16T07:45:50.965670+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":708,"log_dups_size":0,"ondisk_log_size":708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19857408,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":709,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":301}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"32'766","reported_seq":836,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-16T07:45:56.650968+0000","last_change":"2024-04-16T07:45:32.943611+0000","last_active":"2024-04-16T07:45:56.650968+0000","last_peered":"2024-04-16T07:45:56.650968+0000","last_clean":"2024-04-16T07:45:56.650968+0000","last_became_active":"2024-04-16T07:45:32.122235+0000","last_became_peered":"2024-04-16T07:45:32.122235+0000","last_unstale":"2024-04-16T07:45:56.650968+0000","last_undegraded":"2024-04-16T07:45:56.650968+0000","last_fullsized":"2024-04-16T07:45:56.650968+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20643840,"num_objects":315,"num_object_clones":0,"num_object_copies":945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":767,"num_write_kb":34624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":67,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-16T07:45:51.019720+0000","last_change":"2024-04-16T07:45:50.096538+0000","last_active":"2024-04-16T07:45:51.019720+0000","last_peered":"2024-04-16T07:45:51.019720+0000","last_clean":"2024-04-16T07:45:51.019720+0000","last_became_active":"2024-04-16T07:45:50.092108+0000","last_became_peered":"2024-04-16T07:45:50.092108+0000","last_unstale":"2024-04-16T07:45:51.019720+0000","last_undegraded":"2024-04-16T07:45:51.019720+0000","last_fullsized":"2024-04-16T07:45:51.019720+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"32'753","reported_seq":820,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-16T07:45:55.194537+0000","last_change":"2024-04-16T07:45:51.410549+0000","last_active":"2024-04-16T07:45:55.194537+0000","last_peered":"2024-04-16T07:45:55.194537+0000","last_clean":"2024-04-16T07:44:25.799450+0000","last_became_active":"2024-04-16T07:45:51.405214+0000","last_became_peered":"2024-04-16T07:45:51.405214+0000","last_unstale":"2024-04-16T07:45:55.194537+0000","last_undegraded":"2024-04-16T07:45:51.017014+0000","last_fullsized":"2024-04-16T07:45:50.998426+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22544384,"num_objects":344,"num_object_clones":0,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":754,"num_write_kb":35136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":343}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-16T07:45:51.046968+0000","last_change":"2024-04-16T07:45:50.102671+0000","last_active":"2024-04-16T07:45:51.046968+0000","last_peered":"2024-04-16T07:45:51.046968+0000","last_clean":"2024-04-16T07:45:51.046968+0000","last_became_active":"2024-04-16T07:45:50.097645+0000","last_became_peered":"2024-04-16T07:45:50.097645+0000","last_unstale":"2024-04-16T07:45:51.046968+0000","last_undegraded":"2024-04-16T07:45:51.046968+0000","last_fullsized":"2024-04-16T07:45:51.046968+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"32'821","reported_seq":1259,"reported_epoch":32,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-16T07:45:56.804038+0000","last_change":"2024-04-16T07:45:51.572455+0000","last_active":"2024-04-16T07:45:56.804038+0000","last_peered":"2024-04-16T07:45:56.804038+0000","last_clean":"2024-04-16T07:44:25.705296+0000","last_became_active":"2024-04-16T07:45:51.371660+0000","last_became_peered":"2024-04-16T07:45:51.371660+0000","last_unstale":"2024-04-16T07:45:56.804038+0000","last_undegraded":"2024-04-16T07:45:56.804038+0000","last_fullsized":"2024-04-16T07:45:50.964529+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":821,"log_dups_size":0,"ondisk_log_size":821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21561344,"num_objects":329,"num_object_clones":0,"num_object_copies":987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":821,"num_write_kb":36800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":329}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":67,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-16T07:45:50.989557+0000","last_change":"2024-04-16T07:45:50.159085+0000","last_active":"2024-04-16T07:45:50.989557+0000","last_peered":"2024-04-16T07:45:50.989557+0000","last_clean":"2024-04-16T07:45:50.989557+0000","last_became_active":"2024-04-16T07:45:50.153883+0000","last_became_peered":"2024-04-16T07:45:50.153883+0000","last_unstale":"2024-04-16T07:45:50.989557+0000","last_undegraded":"2024-04-16T07:45:50.989557+0000","last_fullsized":"2024-04-16T07:45:50.989557+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"32'728","reported_seq":799,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-16T07:45:55.141413+0000","last_change":"2024-04-16T07:45:32.711215+0000","last_active":"2024-04-16T07:45:55.141413+0000","last_peered":"2024-04-16T07:45:55.141413+0000","last_clean":"2024-04-16T07:45:55.141413+0000","last_became_active":"2024-04-16T07:45:32.130197+0000","last_became_peered":"2024-04-16T07:45:32.130197+0000","last_unstale":"2024-04-16T07:45:55.141413+0000","last_undegraded":"2024-04-16T07:45:55.141413+0000","last_fullsized":"2024-04-16T07:45:55.141413+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:56.287435+0000","last_clean_scrub_stamp":"2024-04-16T07:42:56.287435+0000","objects_scrubbed":0,"log_size":728,"log_dups_size":0,"ondisk_log_size":728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21037056,"num_objects":321,"num_object_clones":0,"num_object_copies":963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":729,"num_write_kb":33600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-16T07:45:51.058516+0000","last_change":"2024-04-16T07:45:50.191150+0000","last_active":"2024-04-16T07:45:51.058516+0000","last_peered":"2024-04-16T07:45:51.058516+0000","last_clean":"2024-04-16T07:45:51.058516+0000","last_became_active":"2024-04-16T07:45:50.184465+0000","last_became_peered":"2024-04-16T07:45:50.184465+0000","last_unstale":"2024-04-16T07:45:51.058516+0000","last_undegraded":"2024-04-16T07:45:51.058516+0000","last_fullsized":"2024-04-16T07:45:51.058516+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:42:44.238048+0000","last_clean_scrub_stamp":"2024-04-16T07:42:44.238048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":219545624,"num_objects":3351,"num_object_clones":0,"num_object_copies":10053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9890,"num_write_kb":415489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":484,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12215,"ondisk_log_size":12215,"up":48,"acting":37,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084313,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":54892,"kb_used_data":18276,"kb_used_omap":0,"kb_used_meta":36608,"kb_avail":94316948,"statfs":{"total":96636764160,"available":96580554752,"internally_reserved":0,"allocated":18714624,"data_stored":294810736,"data_compressed":13996608,"data_compressed_allocated":18423808,"data_compressed_original":294699008,"omap_allocated":0,"internal_metadata":37486592},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705858,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41692,"kb_used_data":14868,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94330148,"statfs":{"total":96636764160,"available":96594071552,"internally_reserved":0,"allocated":15224832,"data_stored":239039576,"data_compressed":11348296,"data_compressed_allocated":14938112,"data_compressed_original":238927872,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280621,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":47260,"kb_used_data":15508,"kb_used_omap":0,"kb_used_meta":31744,"kb_avail":94324580,"statfs":{"total":96636764160,"available":96588369920,"internally_reserved":0,"allocated":15880192,"data_stored":249852496,"data_compressed":11859832,"data_compressed_allocated":15609856,"data_compressed_original":249757696,"omap_allocated":0,"internal_metadata":32505856},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":30,"seq":128849018885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62908,"kb_used_data":22448,"kb_used_omap":0,"kb_used_meta":40448,"kb_avail":94308932,"statfs":{"total":96636764160,"available":96572346368,"internally_reserved":0,"allocated":22986752,"data_stored":363230296,"data_compressed":17244985,"data_compressed_allocated":22700032,"data_compressed_original":363118592,"omap_allocated":0,"internal_metadata":41418752},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:45:58.760 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-backfill 3.b 3.9 2024-04-16T07:45:58.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:58.929+0000 7f066daed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:58.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:58.947+0000 7f066daed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:59.249 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg(s) [3.9] on osd.1 to force-backfill; instructing pg(s) [3.b] on osd.3 to force-backfill; 2024-04-16T07:45:59.249 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-04-16T07:45:59.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:45:59.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:59.570+0000 7fbd30206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:59.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:45:59.581+0000 7fbd30206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:02.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:46:02.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:02.307+0000 7f0a2ec2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:02.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:02.317+0000 7f0a2ec2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:04.269 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 1, 3, 0] 2024-04-16T07:46:04.269 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:46:04.269 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [2, 1, 3, 0] 2024-04-16T07:46:04.269 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-16T07:46:04.269 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:46:04.369 INFO:tasks.ceph.osd.3:Stopped 2024-04-16T07:46:04.690 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:46:04.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:04.809+0000 7fac33f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:04.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:04.816+0000 7fac33f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:09.370 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [2, 1, 0] 2024-04-16T07:46:09.370 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:46:09.370 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-16T07:46:09.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-16T07:46:09.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:09.488+0000 7f29b6b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:09.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:09.498+0000 7f29b6b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:09.514 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-04-16T07:46:09.514 INFO:teuthology.orchestra.run.smithi031.stderr:config get 2024-04-16T07:46:09.514 INFO:teuthology.orchestra.run.smithi031.stderr:config help 2024-04-16T07:46:09.514 INFO:teuthology.orchestra.run.smithi031.stderr:config set ... 2024-04-16T07:46:09.514 INFO:teuthology.orchestra.run.smithi031.stderr:config show 2024-04-16T07:46:09.514 INFO:teuthology.orchestra.run.smithi031.stderr:deep_scrub 2024-04-16T07:46:09.514 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_ops 2024-04-16T07:46:09.514 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_slow_ops 2024-04-16T07:46:09.514 INFO:teuthology.orchestra.run.smithi031.stderr:dump_metrics [] 2024-04-16T07:46:09.514 INFO:teuthology.orchestra.run.smithi031.stderr:dump_ops_in_flight 2024-04-16T07:46:09.515 INFO:teuthology.orchestra.run.smithi031.stderr:dump_pgstate_history 2024-04-16T07:46:09.515 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-04-16T07:46:09.517 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:09.518 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:46:09.518 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-16T07:46:09.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:09.629+0000 7f944955d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:09.638+0000 7f944955d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:09.654 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-04-16T07:46:09.654 INFO:teuthology.orchestra.run.smithi031.stderr:config get 2024-04-16T07:46:09.654 INFO:teuthology.orchestra.run.smithi031.stderr:config help 2024-04-16T07:46:09.654 INFO:teuthology.orchestra.run.smithi031.stderr:config set ... 2024-04-16T07:46:09.654 INFO:teuthology.orchestra.run.smithi031.stderr:config show 2024-04-16T07:46:09.655 INFO:teuthology.orchestra.run.smithi031.stderr:deep_scrub 2024-04-16T07:46:09.655 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_ops 2024-04-16T07:46:09.655 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_slow_ops 2024-04-16T07:46:09.655 INFO:teuthology.orchestra.run.smithi031.stderr:dump_metrics [] 2024-04-16T07:46:09.655 INFO:teuthology.orchestra.run.smithi031.stderr:dump_ops_in_flight 2024-04-16T07:46:09.655 INFO:teuthology.orchestra.run.smithi031.stderr:dump_pgstate_history 2024-04-16T07:46:09.655 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-04-16T07:46:09.658 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:09.658 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:46:09.658 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-16T07:46:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:09.770+0000 7f93acaed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:09.779+0000 7f93acaed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:09.798 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-04-16T07:46:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:config get 2024-04-16T07:46:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:config help 2024-04-16T07:46:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:config set ... 2024-04-16T07:46:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:config show 2024-04-16T07:46:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:deep_scrub 2024-04-16T07:46:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_ops 2024-04-16T07:46:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:dump_historic_slow_ops 2024-04-16T07:46:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:dump_metrics [] 2024-04-16T07:46:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:dump_ops_in_flight 2024-04-16T07:46:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:dump_pgstate_history 2024-04-16T07:46:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-04-16T07:46:09.801 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:09.802 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:46:09.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-16T07:46:09.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:09.912+0000 7f80fb22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:09.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:09.922+0000 7f80fb22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:31.338 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:46:31.338 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:46:31.343 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:46:31.344 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:46:31.344 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:46:31.345 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:46:33.345 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:46:33.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:33.490+0000 7fbcef79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:33.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:33.499+0000 7fbcef79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:33.970 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:46:33.974 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:46:33.974 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:46:35.976 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:46:36.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:36.144+0000 7f34af35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:36.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:36.154+0000 7f34af35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:36.345 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [2, 1, 0] 2024-04-16T07:46:36.345 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:46:36.345 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-04-16T07:46:36.346 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-16T07:46:36.346 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-16T07:46:36.348 INFO:tasks.ceph.osd.3:Started 2024-04-16T07:46:36.349 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:46:36.478 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:46:36.481 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:46:36.483 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:46:36.483 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:46:36.484 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:36.484 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:46:38.485 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:46:38.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:38.641+0000 7f355ae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:38.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:38.650+0000 7f355ae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:39.080 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:46:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:46:39.085 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:46:41.087 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:46:41.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:41.236+0000 7f7d4e0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:41.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:41.246+0000 7f7d4e0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:41.485 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:46:41.502 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:46:41.508 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:46:41.508 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:46:41.621 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:46:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:41.625 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:46:43.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:46:43.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:43.668+0000 7f76a16ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:43.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:43.677+0000 7f76a16ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:43.981 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:46:43.984 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:46:43.985 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:46:45.986 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:46:46.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:46.136+0000 7f9bd5f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:46.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:46.144+0000 7f9bd5f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:46.440 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:46:46.443 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:46:46.444 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:46:46.626 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:46:46.753 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:46:46.757 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:46.758 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:46:48.445 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:46:48.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:48.614+0000 7f002cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:48.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:48.622+0000 7f002cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:48.982 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:46:48.986 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:46:48.987 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:46:50.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:46:51.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:51.160+0000 7fbeb0e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:51.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:51.169+0000 7fbeb0e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:51.759 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:46:51.868 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-16T07:46:51.868 INFO:teuthology.orchestra.run.smithi094.stdout: "ops_in_flight": [ 2024-04-16T07:46:51.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:46:51.869 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:46:51.869 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 37, 2024-04-16T07:46:51.869 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:46:51.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:46:51.869 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:51.869 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:50.280293+0000" 2024-04-16T07:46:51.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.869 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:46:51.869 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:51.869 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:50.280318+0000" 2024-04-16T07:46:51.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.869 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:46:51.869 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:51.869 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:50.283627+0000" 2024-04-16T07:46:51.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.870 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:46:51.870 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:51.870 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:50.285845+0000" 2024-04-16T07:46:51.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.870 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:46:51.870 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:51.870 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:50.319782+0000" 2024-04-16T07:46:51.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.870 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:46:51.870 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:51.870 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:50.320400+0000" 2024-04-16T07:46:51.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.870 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:46:51.870 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:51.871 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:51.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:46:51.871 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:51.871 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:50.998875+0000" 2024-04-16T07:46:51.871 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:46:51.871 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:46:51.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:46:51.871 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:46:51.871 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 41, 2024-04-16T07:46:51.871 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:46:51.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:46:51.872 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:51.872 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:50.300145+0000" 2024-04-16T07:46:51.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.872 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:46:51.872 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:51.872 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:50.300161+0000" 2024-04-16T07:46:51.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.872 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:46:51.872 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:51.872 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:50.302458+0000" 2024-04-16T07:46:51.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.872 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:46:51.872 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:51.872 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:51.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.873 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:46:51.873 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:51.873 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:50.320523+0000" 2024-04-16T07:46:51.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.873 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:46:51.873 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:51.873 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:50.321754+0000" 2024-04-16T07:46:51.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.873 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:46:51.873 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:51.873 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:51.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:46:51.873 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:51.873 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:51.003723+0000" 2024-04-16T07:46:51.873 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:46:51.874 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:46:51.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-16T07:46:51.874 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client_request", 2024-04-16T07:46:51.874 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 44, 2024-04-16T07:46:51.874 INFO:teuthology.orchestra.run.smithi094.stdout: "detail": { 2024-04-16T07:46:51.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:46:51.874 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:51.874 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:50.311380+0000" 2024-04-16T07:46:51.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.874 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:46:51.874 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:51.874 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:50.311396+0000" 2024-04-16T07:46:51.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.875 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:46:51.875 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:51.875 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:50.312962+0000" 2024-04-16T07:46:51.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.875 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:46:51.875 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:51.875 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:51.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.875 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:46:51.875 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:51.875 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:50.322993+0000" 2024-04-16T07:46:51.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.876 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:46:51.876 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:51.876 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:50.324805+0000" 2024-04-16T07:46:51.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.876 INFO:teuthology.orchestra.run.smithi094.stdout: "blocking_event": { 2024-04-16T07:46:51.876 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:51.876 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:51.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-16T07:46:51.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time_event": { 2024-04-16T07:46:51.876 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:51.876 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-04-16T07:46:51.026860+0000" 2024-04-16T07:46:51.876 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:46:51.876 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:46:51.876 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-16T07:46:51.877 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-04-16T07:46:51.877 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-16T07:46:53.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:46:53.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:53.525+0000 7f4279c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:53.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:53.534+0000 7f4279c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:56.047 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:46:56.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:56.198+0000 7fdbb479d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:56.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:56.207+0000 7fdbb479d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:56.878 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 1, 0, 3] 2024-04-16T07:46:56.879 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:46:56.879 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [2, 1, 0, 3] 2024-04-16T07:46:56.879 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-16T07:46:56.879 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:46:56.971 INFO:tasks.ceph.osd.3:Stopped 2024-04-16T07:46:58.780 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:46:58.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:58.905+0000 7f91238ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:58.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:46:58.912+0000 7f91238ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:01.972 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [2, 1, 0] 2024-04-16T07:47:01.972 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:47:01.972 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-04-16T07:47:01.972 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-16T07:47:01.972 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-16T07:47:01.975 INFO:tasks.ceph.osd.3:Started 2024-04-16T07:47:01.975 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:47:02.090 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:47:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:47:02.093 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:47:07.094 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:47:07.205 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:47:07.208 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:47:07.209 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:47:09.953 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:47:09.958 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:47:09.958 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:47:11.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-04-16T07:47:11.959 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:47:12.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:47:12.090+0000 7f702d4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:12.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:47:12.096+0000 7f702d4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:12.210 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:47:12.318 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:47:12.321 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:47:12.321 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:47:17.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~6s 2024-04-16T07:47:17.322 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:47:17.435 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:47:17.438 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:47:17.439 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:47:22.440 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:47:22.557 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:47:22.562 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:47:22.562 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:47:22.697 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~11s 2024-04-16T07:47:27.563 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:47:27.677 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:47:27.680 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:47:27.681 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:47:28.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~17s 2024-04-16T07:47:32.682 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:47:32.794 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:47:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:47:32.798 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:47:33.706 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~22s 2024-04-16T07:47:36.990 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:47:36.995 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:47:36.995 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:47:37.799 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:47:37.904 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:47:37.907 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:47:37.908 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:47:38.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:47:39.006 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Removed 8646 objects 2024-04-16T07:47:39.006 INFO:tasks.radosbench.radosbench.0.smithi031.stdout:Clean up completed and total clean up time :126.539 2024-04-16T07:47:39.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:47:39.113+0000 7f1094b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:39.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:47:39.119+0000 7f1094b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:39.185 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:47:39.188 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:47:39.189 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:47:39.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~28s 2024-04-16T07:47:40.806 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-04-16T07:47:40.806 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-04-16T07:47:40.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:47:40.932+0000 7f20de42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:40.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:47:40.945+0000 7f20de42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:41.189 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:47:41.194 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_0' removed 2024-04-16T07:47:41.213 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-16T07:47:41.223 INFO:tasks.thrashosds:joining thrashosds 2024-04-16T07:47:41.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:47:41.329+0000 7fe96ad9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:41.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:47:41.336+0000 7fe96ad9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:41.470 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:47:41.473 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:47:41.473 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:47:42.908 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:47:43.013 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:47:43.016 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:47:43.017 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:47:44.715 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~33s 2024-04-16T07:47:48.018 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:47:48.132 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:47:48.135 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:47:48.135 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:47:50.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~39s 2024-04-16T07:47:53.136 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:47:53.250 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:47:53.254 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:47:53.254 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:47:55.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~44s 2024-04-16T07:47:58.255 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:47:58.365 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:47:58.369 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:47:58.369 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:48:01.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~50s 2024-04-16T07:48:03.370 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:48:03.484 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:48:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:03.488 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:48:06.733 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~55s 2024-04-16T07:48:08.489 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:48:08.603 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:48:08.607 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:08.607 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:48:12.237 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~61s 2024-04-16T07:48:13.608 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:48:13.721 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:48:13.725 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:13.725 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:48:17.742 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~66s 2024-04-16T07:48:18.726 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:48:18.842 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:48:18.845 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:18.845 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:48:23.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~72s 2024-04-16T07:48:23.847 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:48:23.961 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:48:23.964 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:23.964 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:48:28.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~77s 2024-04-16T07:48:28.966 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:48:29.079 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:48:29.083 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:29.083 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:48:34.084 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:48:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:48:34.201 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:34.202 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:48:34.258 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~83s 2024-04-16T07:48:37.241 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:48:37.242+0000 7ff850ff1640 -1 calc_pg_upmaps abort due to max <= 0 2024-04-16T07:48:39.203 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:48:39.311 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:48:39.314 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:39.314 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:48:39.762 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~88s 2024-04-16T07:48:44.316 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:48:44.429 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:48:44.432 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:44.432 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:48:45.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~94s 2024-04-16T07:48:49.434 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:48:49.547 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:48:49.550 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:49.550 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:48:50.771 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~99s 2024-04-16T07:48:54.551 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:48:54.665 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:48:54.668 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:54.669 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:48:56.276 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~105s 2024-04-16T07:48:59.670 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:48:59.783 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:48:59.787 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:59.787 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:49:01.781 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~110s 2024-04-16T07:49:04.788 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:49:04.902 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:49:04.905 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:49:04.905 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:49:07.286 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~116s 2024-04-16T07:49:09.906 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:49:10.020 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:49:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:49:10.024 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:49:12.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~121s 2024-04-16T07:49:15.024 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:49:15.137 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:49:15.140 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:49:15.141 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:49:18.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~127s 2024-04-16T07:49:20.141 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:49:20.255 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:49:20.258 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:49:20.259 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:49:23.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~132s 2024-04-16T07:49:25.260 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:49:25.375 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:49:25.378 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:49:25.378 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:49:29.307 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~138s 2024-04-16T07:49:30.379 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:49:30.493 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:49:30.497 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:49:30.497 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:49:34.811 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~143s 2024-04-16T07:49:35.498 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:49:35.613 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:49:35.616 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:49:35.616 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:49:37.982 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:49:37.982+0000 7ff850ff1640 -1 calc_pg_upmaps abort due to max <= 0 2024-04-16T07:49:40.316 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~149s 2024-04-16T07:49:40.618 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:49:40.731 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:49:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:49:40.735 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:49:45.736 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:49:45.821 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~154s 2024-04-16T07:49:45.849 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:49:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:49:45.853 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:49:50.854 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:49:50.967 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:49:50.971 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:49:50.971 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:49:51.325 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~160s 2024-04-16T07:49:55.972 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:49:56.088 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:49:56.092 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:49:56.093 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:49:56.828 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~165s 2024-04-16T07:50:01.093 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:50:01.209 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:50:01.212 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:50:01.212 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:50:02.332 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~171s 2024-04-16T07:50:06.214 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:50:06.328 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:50:06.331 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:50:06.332 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:50:07.837 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~176s 2024-04-16T07:50:11.333 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:50:11.448 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:50:11.451 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:50:11.451 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:50:13.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~182s 2024-04-16T07:50:16.452 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:50:16.567 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:50:16.570 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:50:16.570 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:50:18.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~187s 2024-04-16T07:50:21.571 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:50:21.687 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:50:21.691 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:50:21.691 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:50:24.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~193s 2024-04-16T07:50:26.693 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:50:26.806 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:50:26.809 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:50:26.810 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:50:29.855 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~198s 2024-04-16T07:50:31.810 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:50:31.924 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:50:31.927 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:50:31.928 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:50:35.359 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~204s 2024-04-16T07:50:36.929 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:50:37.043 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:50:37.046 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:50:37.046 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:50:37.996 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:50:37.996+0000 7ff850ff1640 -1 calc_pg_upmaps abort due to max <= 0 2024-04-16T07:50:40.864 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~209s 2024-04-16T07:50:42.048 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:50:42.163 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:50:42.165 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:50:42.166 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:50:46.369 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~215s 2024-04-16T07:50:47.167 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:50:47.280 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:50:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:50:47.283 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:50:51.873 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~220s 2024-04-16T07:50:52.284 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:50:52.398 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:50:52.401 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:50:52.401 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:50:57.378 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~226s 2024-04-16T07:50:57.403 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:50:57.516 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:50:57.519 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:50:57.519 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:51:02.520 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:51:02.634 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:51:02.637 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:51:02.638 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:51:02.882 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~231s 2024-04-16T07:51:07.639 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:51:07.754 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:51:07.757 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:51:07.757 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:51:08.386 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~237s 2024-04-16T07:51:12.758 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:51:12.871 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:51:12.874 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:51:12.874 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:51:13.891 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~242s 2024-04-16T07:51:17.875 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:51:17.990 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:51:17.993 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:51:17.993 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:51:19.395 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~248s 2024-04-16T07:51:22.994 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:51:23.108 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:51:23.111 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:51:23.112 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:51:24.900 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~253s 2024-04-16T07:51:28.112 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:51:28.227 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:51:28.230 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:51:28.230 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:51:30.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~259s 2024-04-16T07:51:33.231 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:51:33.345 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:51:33.349 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:51:33.349 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:51:35.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~264s 2024-04-16T07:51:37.999 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:51:37.999+0000 7ff850ff1640 -1 calc_pg_upmaps abort due to max <= 0 2024-04-16T07:51:38.350 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:51:38.463 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:51:38.466 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:51:38.466 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:51:41.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~270s 2024-04-16T07:51:43.467 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:51:43.581 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:51:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:51:43.584 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:51:46.917 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~275s 2024-04-16T07:51:48.585 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:51:48.701 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:51:48.704 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:51:48.705 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:51:52.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~281s 2024-04-16T07:51:53.705 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:51:53.820 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:51:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:51:53.823 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:51:57.927 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~286s 2024-04-16T07:51:58.825 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:51:58.938 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:51:58.941 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:51:58.942 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:52:03.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~292s 2024-04-16T07:52:03.943 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:52:04.057 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:52:04.059 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:52:04.060 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:52:08.936 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~297s 2024-04-16T07:52:09.061 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:52:09.175 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:52:09.177 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:52:09.178 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:52:14.179 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:52:14.293 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:52:14.296 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:52:14.296 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:52:14.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~303s 2024-04-16T07:52:14.441 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-04-16T07:52:14.944 INFO:tasks.ceph.osd.0:Sent signal 15 2024-04-16T07:52:14.945 INFO:tasks.ceph.osd.1:Sent signal 15 2024-04-16T07:52:14.945 INFO:tasks.ceph.osd.2:Sent signal 15 2024-04-16T07:52:14.946 INFO:tasks.ceph.mon.a:Sent signal 15 2024-04-16T07:52:14.946 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-04-16T07:52:14.946 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-04-16T07:52:14.947+0000 7f062c5ff640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 29913) UID: 0 2024-04-16T07:52:14.947 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-04-16T07:52:14.947+0000 7f062c5ff640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-16T07:52:14.947 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-04-16T07:52:14.946+0000 7ff8b11f7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 29258) UID: 0 2024-04-16T07:52:19.297 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:52:19.409 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:52:19.412 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:52:19.413 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:52:24.413 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:52:24.528 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:52:24.530 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:52:24.531 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:52:29.531 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:52:29.645 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:52:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:52:29.649 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:52:34.650 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:52:34.765 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:52:34.768 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:52:34.768 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:52:39.769 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:52:39.883 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:52:39.886 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:52:39.887 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:52:44.888 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:52:45.001 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:52:45.005 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:52:45.005 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:52:50.006 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:52:50.120 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:52:50.123 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:52:50.123 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:52:55.124 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:52:55.238 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:52:55.241 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:52:55.241 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:53:00.243 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:53:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:53:00.359 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:53:00.360 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:53:05.361 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:53:05.475 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:53:05.479 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:53:05.479 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:53:10.480 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:53:10.593 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:53:10.598 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:53:10.600 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1461, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 560, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 3122, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2028, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.3 restart 2024-04-16T07:53:10.600 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1332, in do_thrash self._do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1461, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 560, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 3122, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2028, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.3 restart 2024-04-16T07:53:10.600 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-16T07:53:10.600 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-16T07:53:10.601 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:53:10.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:53:10.727+0000 7f1bc7e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:55:10.642 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T07:55:10.644 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-04-16T07:55:10.645 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-16T07:55:10.655 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T07:55:10.656 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:55:10.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-16T07:55:10.780+0000 7fa0a9aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:57:10.696 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T07:57:10.697 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T07:57:10.699 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-16T07:57:10.699 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-16T07:57:10.699 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-16T07:57:10.699 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:57:10.797 INFO:tasks.ceph.osd.0:Stopped 2024-04-16T07:57:10.797 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-16T07:57:10.798 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:57:10.893 INFO:tasks.ceph.osd.1:Stopped 2024-04-16T07:57:10.893 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-16T07:57:10.893 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:57:10.971 INFO:tasks.ceph.osd.2:Stopped 2024-04-16T07:57:10.972 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-16T07:57:10.972 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:57:10.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:57:10.972 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi094 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log' 2024-04-16T07:57:10.973 INFO:tasks.ceph.osd.3:Stopped 2024-04-16T07:57:10.973 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-16T07:57:10.973 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-16T07:57:10.973 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:57:10.973 INFO:tasks.ceph.mgr.x:Stopped 2024-04-16T07:57:10.973 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-16T07:57:10.973 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-16T07:57:10.973 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:57:10.974 INFO:tasks.ceph.mon.a:Stopped 2024-04-16T07:57:10.974 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-16T07:57:10.974 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-16T07:57:11.014 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi031.front.sepia.ceph.com 2024-04-16T07:57:11.014 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-16T07:57:11.124 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi031.front.sepia.ceph.com 2024-04-16T07:57:11.125 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-16T07:57:11.215 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi094.front.sepia.ceph.com 2024-04-16T07:57:11.215 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-16T07:57:11.354 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi094.front.sepia.ceph.com 2024-04-16T07:57:11.355 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-16T07:57:11.420 INFO:tasks.ceph:Archiving mon data... 2024-04-16T07:57:11.421 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657957/data/mon.a.tgz 2024-04-16T07:57:11.421 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-04-16T07:57:11.436 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.JnfzDHHfAx 2024-04-16T07:57:11.436 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.JnfzDHHfAx 2024-04-16T07:57:11.579 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.JnfzDHHfAx 2024-04-16T07:57:11.665 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.JnfzDHHfAx is 304KB 2024-04-16T07:57:11.732 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.JnfzDHHfAx 2024-04-16T07:57:11.746 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-16T07:57:11.746 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T07:57:11.788 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T07:57:11.870 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-16T07:57:11.888 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-16T07:57:11.888 INFO:tasks.ceph:Archiving crash dumps... 2024-04-16T07:57:11.890 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657957/remote/smithi031/crash 2024-04-16T07:57:11.891 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-16T07:57:11.939 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657957/remote/smithi094/crash 2024-04-16T07:57:11.940 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-16T07:57:11.984 INFO:tasks.ceph:Compressing logs... 2024-04-16T07:57:11.984 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T07:57:11.987 DEBUG:teuthology.orchestra.run.smithi094:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T07:57:12.012 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34632.log 2024-04-16T07:57:12.012 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33503.log 2024-04-16T07:57:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35366.log 2024-04-16T07:57:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34632.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34632.log.gz 2024-04-16T07:57:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33503.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-16T07:57:12.059 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33503.log.gz 2024-04-16T07:57:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31810.log 2024-04-16T07:57:12.059 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35366.log.gz 2024-04-16T07:57:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32597.log 2024-04-16T07:57:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34724.log 2024-04-16T07:57:12.059 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31810.log.gz 2024-04-16T07:57:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30592.log 2024-04-16T07:57:12.060 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32597.log.gz 2024-04-16T07:57:12.060 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31220.log 2024-04-16T07:57:12.060 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34724.log.gz 2024-04-16T07:57:12.060 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33901.log 2024-04-16T07:57:12.060 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30592.log.gz 2024-04-16T07:57:12.060 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34816.log 2024-04-16T07:57:12.060 INFO:teuthology.orchestra.run.smithi031.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.31220.log.gz 2024-04-16T07:57:12.060 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33336.log 2024-04-16T07:57:12.060 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33901.log.gz 2024-04-16T07:57:12.060 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32967.log 2024-04-16T07:57:12.060 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34816.log.gz 2024-04-16T07:57:12.060 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36352.log 2024-04-16T07:57:12.060 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33336.log.gz 2024-04-16T07:57:12.060 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32376.log 2024-04-16T07:57:12.061 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32967.log.gz 2024-04-16T07:57:12.061 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34116.log 2024-04-16T07:57:12.061 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36352.log.gz 2024-04-16T07:57:12.061 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31894.log 2024-04-16T07:57:12.061 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32376.log.gz 2024-04-16T07:57:12.061 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35811.log 2024-04-16T07:57:12.061 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34116.log.gz 2024-04-16T07:57:12.061 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36497.log 2024-04-16T07:57:12.061 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31894.log.gz 2024-04-16T07:57:12.061 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35624.log 2024-04-16T07:57:12.061 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35811.log.gz 2024-04-16T07:57:12.061 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35666.log 2024-04-16T07:57:12.061 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36497.log.gz 2024-04-16T07:57:12.062 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36450.log 2024-04-16T07:57:12.062 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35624.log.gz 2024-04-16T07:57:12.062 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30500.log 2024-04-16T07:57:12.062 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35666.log.gz 2024-04-16T07:57:12.062 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30844.log 2024-04-16T07:57:12.062 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36450.log.gz 2024-04-16T07:57:12.062 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30458.log 2024-04-16T07:57:12.062 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30500.log.gz 2024-04-16T07:57:12.062 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30550.log 2024-04-16T07:57:12.062 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30844.log.gz 2024-04-16T07:57:12.062 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30150.log 2024-04-16T07:57:12.062 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30458.log.gz 2024-04-16T07:57:12.062 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31576.log 2024-04-16T07:57:12.062 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30550.log.gz 2024-04-16T07:57:12.063 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33016.log 2024-04-16T07:57:12.063 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30150.log.gz 2024-04-16T07:57:12.063 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35851.log 2024-04-16T07:57:12.063 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31576.log.gz 2024-04-16T07:57:12.063 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36021.log 2024-04-16T07:57:12.063 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33016.log.gz 2024-04-16T07:57:12.063 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31073.log 2024-04-16T07:57:12.063 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35851.log.gz 2024-04-16T07:57:12.063 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31536.log 2024-04-16T07:57:12.063 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36021.log.gz 2024-04-16T07:57:12.063 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31073.log: gzip 50.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.31073.log.gz -- 2024-04-16T07:57:12.063 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.36256.log 2024-04-16T07:57:12.063 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30902.log 2024-04-16T07:57:12.063 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31536.log.gz 2024-04-16T07:57:12.063 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30634.log 2024-04-16T07:57:12.064 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36256.log.gz 2024-04-16T07:57:12.064 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33723.log 2024-04-16T07:57:12.064 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30902.log.gz 2024-04-16T07:57:12.064 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32499.log 2024-04-16T07:57:12.064 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30634.log.gz 2024-04-16T07:57:12.064 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33723.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28858.log 2024-04-16T07:57:12.064 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33723.log.gz 2024-04-16T07:57:12.064 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32235.log 2024-04-16T07:57:12.064 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32499.log.gz 2024-04-16T07:57:12.064 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.tmp-client.admin.28858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33065.log 2024-04-16T07:57:12.064 INFO:teuthology.orchestra.run.smithi031.stderr: 68.3% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28858.log.gz 2024-04-16T07:57:12.064 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35764.log 2024-04-16T07:57:12.064 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32235.log.gz 2024-04-16T07:57:12.064 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35279.log 2024-04-16T07:57:12.064 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33065.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33065.log.gz 2024-04-16T07:57:12.065 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30733.log 2024-04-16T07:57:12.065 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35764.log.gz 2024-04-16T07:57:12.065 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35279.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-16T07:57:12.065 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35279.log.gz 2024-04-16T07:57:12.065 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30736.log 2024-04-16T07:57:12.065 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30733.log.gz 2024-04-16T07:57:12.065 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33852.log 2024-04-16T07:57:12.065 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36633.log 2024-04-16T07:57:12.065 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30736.log.gz 2024-04-16T07:57:12.065 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33683.log 2024-04-16T07:57:12.065 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-16T07:57:12.065 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33852.log.gz 2024-04-16T07:57:12.065 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36115.log 2024-04-16T07:57:12.065 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36633.log.gz 2024-04-16T07:57:12.065 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36068.log 2024-04-16T07:57:12.066 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33683.log.gz 2024-04-16T07:57:12.066 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29994.log 2024-04-16T07:57:12.066 INFO:teuthology.orchestra.run.smithi031.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.36115.log.gz 2024-04-16T07:57:12.066 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34408.log 2024-04-16T07:57:12.066 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36068.log.gz 2024-04-16T07:57:12.066 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.29994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34368.log 2024-04-16T07:57:12.066 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29994.log.gz 2024-04-16T07:57:12.066 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30903.log 2024-04-16T07:57:12.066 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34408.log.gz 2024-04-16T07:57:12.066 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30904.log 2024-04-16T07:57:12.066 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34368.log.gz 2024-04-16T07:57:12.066 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32457.log 2024-04-16T07:57:12.066 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30903.log.gz 2024-04-16T07:57:12.067 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.30904.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34540.log 2024-04-16T07:57:12.067 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30904.log.gz 2024-04-16T07:57:12.067 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32737.log 2024-04-16T07:57:12.067 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32457.log.gz 2024-04-16T07:57:12.067 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36162.log 2024-04-16T07:57:12.067 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34540.log.gz 2024-04-16T07:57:12.067 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34288.log 2024-04-16T07:57:12.067 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32737.log.gz 2024-04-16T07:57:12.067 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32688.log 2024-04-16T07:57:12.067 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36162.log.gz 2024-04-16T07:57:12.067 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29915.log 2024-04-16T07:57:12.067 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34288.log.gz 2024-04-16T07:57:12.067 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31032.log 2024-04-16T07:57:12.067 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32688.log.gz 2024-04-16T07:57:12.067 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.29915.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-16T07:57:12.068 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29915.log.gz 2024-04-16T07:57:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34770.log 2024-04-16T07:57:12.068 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31032.log.gz 2024-04-16T07:57:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34072.log 2024-04-16T07:57:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31726.log 2024-04-16T07:57:12.068 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34770.log.gz 2024-04-16T07:57:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31072.log 2024-04-16T07:57:12.068 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34072.log.gz 2024-04-16T07:57:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31726.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31726.log.gz 2024-04-16T07:57:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30735.log 2024-04-16T07:57:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34586.log 2024-04-16T07:57:12.068 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31072.log.gz 2024-04-16T07:57:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35214.log 2024-04-16T07:57:12.068 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30735.log.gz 2024-04-16T07:57:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33594.log 2024-04-16T07:57:12.069 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34586.log.gz 2024-04-16T07:57:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35533.log 2024-04-16T07:57:12.069 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35214.log.gz 2024-04-16T07:57:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32193.log 2024-04-16T07:57:12.069 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33594.log.gz 2024-04-16T07:57:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36401.log 2024-04-16T07:57:12.069 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35533.log.gz 2024-04-16T07:57:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32779.log 2024-04-16T07:57:12.069 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32193.log.gz 2024-04-16T07:57:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32097.log 2024-04-16T07:57:12.069 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36401.log.gz 2024-04-16T07:57:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33763.log 2024-04-16T07:57:12.069 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32779.log.gz 2024-04-16T07:57:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34242.log 2024-04-16T07:57:12.070 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32097.log.gz 2024-04-16T07:57:12.070 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33763.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33763.log.gz 2024-04-16T07:57:12.070 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35715.log 2024-04-16T07:57:12.070 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32828.log 2024-04-16T07:57:12.070 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34242.log.gz 2024-04-16T07:57:12.070 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35715.log: gzip 50.4% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.35715.log.gz -- 2024-04-16T07:57:12.070 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.31219.log 2024-04-16T07:57:12.070 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31240.log 2024-04-16T07:57:12.070 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32828.log.gz 2024-04-16T07:57:12.070 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31219.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31219.log.gz 2024-04-16T07:57:12.070 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31852.log 2024-04-16T07:57:12.070 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31768.log 2024-04-16T07:57:12.070 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31240.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31240.log.gz 2024-04-16T07:57:12.070 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31418.log 2024-04-16T07:57:12.071 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31852.log.gz 2024-04-16T07:57:12.071 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33454.log 2024-04-16T07:57:12.071 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31768.log.gz 2024-04-16T07:57:12.071 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35974.log 2024-04-16T07:57:12.071 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31418.log.gz 2024-04-16T07:57:12.071 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32925.log 2024-04-16T07:57:12.071 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33454.log.gz 2024-04-16T07:57:12.071 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30402.log 2024-04-16T07:57:12.071 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35974.log.gz 2024-04-16T07:57:12.071 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31936.log 2024-04-16T07:57:12.071 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32925.log.gz 2024-04-16T07:57:12.071 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34678.log 2024-04-16T07:57:12.071 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30402.log.gz 2024-04-16T07:57:12.071 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31516.log 2024-04-16T07:57:12.072 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31936.log.gz 2024-04-16T07:57:12.072 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33803.log 2024-04-16T07:57:12.072 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34678.log.gz 2024-04-16T07:57:12.072 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35415.log 2024-04-16T07:57:12.072 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31516.log.gz 2024-04-16T07:57:12.072 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33803.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33803.log.gz 2024-04-16T07:57:12.072 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35484.log 2024-04-16T07:57:12.072 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34448.log 2024-04-16T07:57:12.072 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35415.log.gz 2024-04-16T07:57:12.072 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35484.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35484.log.gz 2024-04-16T07:57:12.072 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.32265.log 2024-04-16T07:57:12.072 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34862.log 2024-04-16T07:57:12.072 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34448.log.gz 2024-04-16T07:57:12.072 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.0.32265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35931.log 2024-04-16T07:57:12.072 INFO:teuthology.orchestra.run.smithi031.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.0.32265.log.gz 2024-04-16T07:57:12.073 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33990.log 2024-04-16T07:57:12.073 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34862.log.gz 2024-04-16T07:57:12.073 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35931.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35931.log.gz 2024-04-16T07:57:12.073 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-16T07:57:12.073 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35582.log 2024-04-16T07:57:12.073 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33990.log.gz 2024-04-16T07:57:12.073 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33114.log 2024-04-16T07:57:12.073 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33948.log 2024-04-16T07:57:12.073 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35582.log.gz 2024-04-16T07:57:12.073 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31180.log 2024-04-16T07:57:12.073 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33114.log.gz 2024-04-16T07:57:12.073 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34909.log 2024-04-16T07:57:12.073 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33948.log.gz 2024-04-16T07:57:12.073 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34196.log 2024-04-16T07:57:12.073 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31180.log.gz 2024-04-16T07:57:12.074 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36591.log 2024-04-16T07:57:12.074 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34909.log.gz 2024-04-16T07:57:12.074 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35064.log 2024-04-16T07:57:12.074 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34196.log.gz 2024-04-16T07:57:12.074 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32639.log 2024-04-16T07:57:12.074 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36591.log.gz 2024-04-16T07:57:12.074 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-16T07:57:12.074 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35064.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35064.log.gz 2024-04-16T07:57:12.074 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32639.log: gzip 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32639.log.gz 2024-04-16T07:57:12.074 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33405.log 2024-04-16T07:57:12.074 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34328.log 2024-04-16T07:57:12.074 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33405.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33405.log.gz 2024-04-16T07:57:12.074 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35131.log 2024-04-16T07:57:12.074 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30734.log 2024-04-16T07:57:12.075 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34328.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34328.log.gz 2024-04-16T07:57:12.075 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31577.log 2024-04-16T07:57:12.075 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35131.log.gz 2024-04-16T07:57:12.075 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30178.log 2024-04-16T07:57:12.075 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30734.log.gz 2024-04-16T07:57:12.075 INFO:teuthology.orchestra.run.smithi031.stderr: 88.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-16T07:57:12.075 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36303.log 2024-04-16T07:57:12.075 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31577.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31577.log.gz 2024-04-16T07:57:12.075 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30178.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30178.log.gz 2024-04-16T07:57:12.075 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.35891.log 2024-04-16T07:57:12.075 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36303.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36303.log.gz 2024-04-16T07:57:12.075 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36684.log 2024-04-16T07:57:12.075 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31998.log 2024-04-16T07:57:12.075 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35891.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35891.log.gz 2024-04-16T07:57:12.076 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36714.log 2024-04-16T07:57:12.076 INFO:teuthology.orchestra.run.smithi031.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.36684.log.gz 2024-04-16T07:57:12.076 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31417.log 2024-04-16T07:57:12.076 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31998.log.gz 2024-04-16T07:57:12.076 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32095.log 2024-04-16T07:57:12.076 INFO:teuthology.orchestra.run.smithi031.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.36714.log.gz 2024-04-16T07:57:12.076 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34030.log 2024-04-16T07:57:12.076 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31417.log.gz 2024-04-16T07:57:12.076 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32877.log 2024-04-16T07:57:12.076 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32095.log.gz 2024-04-16T07:57:12.076 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31368.log 2024-04-16T07:57:12.076 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34030.log.gz 2024-04-16T07:57:12.076 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32877.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32877.log.gz 2024-04-16T07:57:12.076 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31684.log 2024-04-16T07:57:12.076 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32096.log 2024-04-16T07:57:12.077 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31368.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31368.log.gz 2024-04-16T07:57:12.077 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33643.log 2024-04-16T07:57:12.077 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31684.log.gz 2024-04-16T07:57:12.077 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34494.log 2024-04-16T07:57:12.077 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32096.log.gz 2024-04-16T07:57:12.077 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33643.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33643.log.gz 2024-04-16T07:57:12.077 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34156.log 2024-04-16T07:57:12.077 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34956.log 2024-04-16T07:57:12.077 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34494.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34494.log.gz 2024-04-16T07:57:12.077 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34156.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.34156.log.gz 2024-04-16T07:57:12.077 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31348.log 2024-04-16T07:57:12.077 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33545.log 2024-04-16T07:57:12.077 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.34956.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34956.log.gz 2024-04-16T07:57:12.077 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31012.log 2024-04-16T07:57:12.078 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31348.log.gz 2024-04-16T07:57:12.078 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36544.log 2024-04-16T07:57:12.078 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33545.log.gz 2024-04-16T07:57:12.078 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32548.log 2024-04-16T07:57:12.078 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.31012.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31012.log.gz 2024-04-16T07:57:12.078 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36544.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36544.log.gz 2024-04-16T07:57:12.078 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36209.log 2024-04-16T07:57:12.078 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32548.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32548.log.gz 2024-04-16T07:57:12.078 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36209.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36209.log.gz 2024-04-16T07:57:12.080 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29378.log 2024-04-16T07:57:12.080 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29338.log 2024-04-16T07:57:12.081 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.29378.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-16T07:57:12.081 INFO:teuthology.orchestra.run.smithi094.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29378.log.gz 2024-04-16T07:57:12.081 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.29338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29458.log 2024-04-16T07:57:12.081 INFO:teuthology.orchestra.run.smithi094.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29338.log.gz 2024-04-16T07:57:12.081 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-16T07:57:12.081 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.29458.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-16T07:57:12.081 INFO:teuthology.orchestra.run.smithi094.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29458.log.gz 2024-04-16T07:57:12.081 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29418.log 2024-04-16T07:57:12.081 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.29418.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29418.log.gz 2024-04-16T07:57:12.121 INFO:teuthology.orchestra.run.smithi094.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-16T07:57:12.204 INFO:teuthology.orchestra.run.smithi031.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-16T07:57:15.636 INFO:teuthology.orchestra.run.smithi094.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-16T07:57:15.802 INFO:teuthology.orchestra.run.smithi031.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-16T07:57:16.375 INFO:teuthology.orchestra.run.smithi094.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-16T07:57:16.377 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-04-16T07:57:16.377 INFO:teuthology.orchestra.run.smithi094.stderr:real 0m4.328s 2024-04-16T07:57:16.377 INFO:teuthology.orchestra.run.smithi094.stderr:user 0m7.660s 2024-04-16T07:57:16.377 INFO:teuthology.orchestra.run.smithi094.stderr:sys 0m0.295s 2024-04-16T07:57:16.464 INFO:teuthology.orchestra.run.smithi031.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-16T07:57:16.466 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-04-16T07:57:16.466 INFO:teuthology.orchestra.run.smithi031.stderr:real 0m4.467s 2024-04-16T07:57:16.466 INFO:teuthology.orchestra.run.smithi031.stderr:user 0m8.135s 2024-04-16T07:57:16.466 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m0.370s 2024-04-16T07:57:16.466 INFO:tasks.ceph:Archiving logs... 2024-04-16T07:57:16.467 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657957/remote/smithi031/log 2024-04-16T07:57:16.468 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T07:57:17.953 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/log/ceph to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657957/remote/smithi094/log 2024-04-16T07:57:17.954 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T07:57:19.212 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T07:57:19.212 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-16T07:57:19.221 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-04-16T07:57:19.221 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-16T07:57:19.221 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T07:57:19.224 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T07:57:19.324 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-16T07:57:19.333 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-16T07:57:19.333 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T07:57:19.336 DEBUG:teuthology.orchestra.run.smithi094:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T07:57:19.350 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-04-16T07:57:19.352 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-04-16T07:57:19.355 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T07:57:19.355 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-04-16T07:57:19.355 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv01.front.sepia.ceph.com 3 6 377 23 +1495us[+1505us] +/- 47ms 2024-04-16T07:57:19.356 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 27 -815us[ -806us] +/- 59ms 2024-04-16T07:57:19.356 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 -1694us[-1684us] +/- 88ms 2024-04-16T07:57:19.356 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:57:19.357 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T07:57:19.357 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-04-16T07:57:19.357 INFO:teuthology.orchestra.run.smithi094.stdout:^* hv01.front.sepia.ceph.com 3 6 377 27 +1535us[+1563us] +/- 47ms 2024-04-16T07:57:19.358 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 23 -761us[ -761us] +/- 59ms 2024-04-16T07:57:19.358 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 -1633us[-1633us] +/- 88ms 2024-04-16T07:57:19.358 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:57:19.358 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-16T07:57:19.402 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-16T07:57:19.402 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-16T07:57:19.449 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T07:57:19.525 DEBUG:teuthology.orchestra.run.smithi094:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T07:57:19.600 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T07:57:19.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:57:19.634 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T07:57:19.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:57:19.667 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-16T07:57:19.676 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-16T07:57:19.720 INFO:teuthology.task.internal:Duration was 1524.001197 seconds 2024-04-16T07:57:19.720 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-16T07:57:19.756 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-16T07:57:19.756 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T07:57:19.759 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T07:57:19.805 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T07:57:19.807 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T07:57:19.961 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-16T07:57:19.961 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2024-04-16T07:57:19.962 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T07:57:19.983 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi094.front.sepia.ceph.com 2024-04-16T07:57:19.983 DEBUG:teuthology.orchestra.run.smithi094:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T07:57:20.011 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-16T07:57:20.011 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T07:57:20.026 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T07:57:20.081 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-16T07:57:20.082 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T07:57:20.096 DEBUG:teuthology.orchestra.run.smithi094:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T07:57:20.225 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-16T07:57:20.235 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-16T07:57:20.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T07:57:20.263 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T07:57:20.291 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-16T07:57:20.301 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T07:57:20.306 DEBUG:teuthology.orchestra.run.smithi094:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T07:57:20.329 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2024-04-16T07:57:20.358 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = core 2024-04-16T07:57:20.381 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T07:57:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:57:20.422 DEBUG:teuthology.orchestra.run.smithi094:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T07:57:20.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:57:20.437 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-16T07:57:20.446 INFO:teuthology.task.internal:Transferring archived files... 2024-04-16T07:57:20.447 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657957/remote/smithi031 2024-04-16T07:57:20.447 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T07:57:20.500 DEBUG:teuthology.misc:Transferring archived files from smithi094:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657957/remote/smithi094 2024-04-16T07:57:20.500 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T07:57:20.537 INFO:teuthology.task.internal:Removing archive directory... 2024-04-16T07:57:20.537 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T07:57:20.539 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T07:57:20.588 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-16T07:57:20.599 INFO:teuthology.task.internal:Not uploading archives. 2024-04-16T07:57:20.599 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-16T07:57:20.632 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-16T07:57:20.632 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T07:57:20.634 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T07:57:20.646 INFO:teuthology.orchestra.run.smithi031.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 07:57 /home/ubuntu/cephtest 2024-04-16T07:57:20.649 INFO:teuthology.orchestra.run.smithi094.stdout: 265594 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 07:57 /home/ubuntu/cephtest 2024-04-16T07:57:20.650 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-16T07:57:20.665 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-16T07:57:20.701 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi031 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-04-16T07:57:20.701 INFO:teuthology.nuke:Checking targets against current locks 2024-04-16T07:57:20.728 DEBUG:teuthology.nuke:shortname: smithi031 2024-04-16T07:57:20.728 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T07:57:20.751 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657957', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 07:23:06.314856', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBS07dOWcaLjUiOU6r19LBrr0T8Y6Zjb8ZCWZeDak9Zu/sx6QisBlA9o2b3LdPcm1uZCdsJBZZe7jXOMEGadN0c='} 2024-04-16T07:57:20.755 DEBUG:teuthology.nuke:shortname: smithi094 2024-04-16T07:57:20.755 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T07:57:20.779 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657957', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 07:23:06.316279', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCB2K1ya0mQ2pOgjC1b+JEgG9txeTVJWCARqu4FLJYKv0kBC4TWbRYstDVBDoII6KaXMqEibABpKWxNkhPZ2Des='} 2024-04-16T07:57:20.807 INFO:teuthology.orchestra.console.smithi031:Power off 2024-04-16T07:57:20.807 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T07:57:20.822 INFO:teuthology.orchestra.console.smithi094:Power off 2024-04-16T07:57:20.822 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T07:57:20.834 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2024-04-16T07:57:20.834 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T07:57:20.843 DEBUG:teuthology.orchestra.console.smithi094:power off output: Chassis Power Control: Down/Off 2024-04-16T07:57:20.843 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T07:57:20.856 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-04-16T07:57:20.865 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2024-04-16T07:57:24.858 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T07:57:24.867 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T07:57:24.969 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-04-16T07:57:24.975 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2024-04-16T07:57:28.972 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T07:57:28.980 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T07:57:29.083 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2024-04-16T07:57:29.090 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is off 2024-04-16T07:57:29.184 INFO:teuthology.orchestra.console.smithi031:Power off completed 2024-04-16T07:57:29.191 INFO:teuthology.orchestra.console.smithi094:Power off completed 2024-04-16T07:57:29.295 INFO:teuthology.run:Summary data: description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} duration: 1524.001196861267 failure_reason: 'Command failed on smithi031 with status 124: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json''' flavor: crimson owner: scheduled_matan@teuthology status: fail success: false 2024-04-16T07:57:29.296 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T07:57:29.386 INFO:teuthology.run:FAIL